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Abstract 

For  the  past  50  years,  the  military’s  use  of  space  for  our  national  defense  has 
increased  exponentially.  The  use  of  space  has  increased  so  much  that  recent  events  have 
led  to  the  approval  for  most  space  Major  Defense  Acquisition  Programs  to  fall  under  their 
own  process  of  oversight  to  track  and  monitor  these  programs.  The  largest  reason  for  this 
change  is  due  to  the  difference  in  spending  profiles  and  current  acquisition  regulations 
that  are  not  structured  to  meet  these  space  expenditure  plans.  The  key  problem  is  no  one 
knows,  for  sure,  how  much  the  oversight  process  actually  costs  and  if  one  form  of 
oversight  is  actually  statistically  better  than  the  other.  If  the  other  processes  are  better, 
what  actually  drives  the  cost  for  their  oversight? 

This  thesis  will  provide  a  foundation  and  potential  cost  saving  recommendations 
that  would  benefit  the  Department  of  Defense  in  most  of  the  acquisition  programs  it 
monitors.  The  cost  of  oversight  will  be  forecasted  based  on  a  panel  of  experts  in  the 
field,  using  the  Delphi  Methodology.  These  costs  will  then  compare  with  other  oversight 
processes  for  the  Department  of  Defense  Directive  5000  and  the  Command,  Control, 
Communication,  and  Intelligence  new  virtual  process,  to  discover  where  the  statistical 
differences  are  in  the  cost  of  oversight.  The  total  costs  for  all  three  oversight  processes 
will  then  provide  insight  on  where  the  largest  cost  benefits  appear  to  be,  based  on  data 
collected,  and  recommendations  will  develop  a  future  track  for  the  next  generation  of 
oversight  processes. 
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COMPARATIVE  ANALYSIS  ON  THE  COST  OF  OVERSIGHT 


FOR  THE  NEW  NATIONAL  SECURITY  SPACE  ACQUISITION 
POLICY— A  DELPHI  METHOD  APPROACH 


1.0  Introduction 


1.1  Overview 

Space  has  been  an  important  part  of  today’s  military  operations.  It  is  hard  to 
imagine  a  military  engagement  in  OPERATION  ENDURING  FREEDOM  or 
OPERATION  IRAQI  FREEDOM  that  did  not  involve  military  technologies  in  space. 
From  the  collection  of  intelligence,  the  communication  of  information,  the  positioning  of 
ground  troops,  the  target  selection,  the  bomb  positioning  via  global  positioning  guidance 
systems,  and  the  post  impact  analysis,  space  applications  are  involved  in  every  step. 
Moving  into  the  future,  space  will  be  even  more  necessary  to  ensure  the  United  States’ 
national  security.  Increases  in  the  need  for  communication  and  intelligence  satellites  and 
the  need  for  the  Ballistic  Missile  Defense  System  places  additional  stresses  on  an  already 
stretched  budget.  There  is  a  need  to  make  sure  every  dollar  is  being  spent  properly  and 
prudently. 

With  the  creation  of  a  new  process  for  all  space  acquisition  programs,  there  is 
hope  that  the  needed  space  systems  will  be  procured  cheaper  and  easier  through  a  new 
acquisition  process.  The  process  will  decentralize  the  acquisition  procurement  process  by 
moving  more  decisions  to  the  lower  levels  of  government.  The  problem  with  the  new 
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oversight  process  is  that  no  one  knows  if  it  actually  saves  money  and  if  oversight  is  better 
when  it  is  centralized  or  decentralized. 

This  thesis  will  use  the  Delphi  Method  of  forecasting  to  estimate  the  cost  of 
oversight  for  space  acquisition  in  the  new  National  Security  Space  Acquisition  Policy 
(NSSAP)  03-01.  By  using  experts  in  the  space  acquisition  field,  an  estimate  of  oversight 
costs  will  be  established  and  compared  to  other  acquisition  processes.  The  goal  will  be  to 
evaluate  if  the  perception  of  new  streamlining  reforms  are  taking  affect  and  benefiting  the 
acquisition  oversight  processes  as  well  as  identifying  the  five  main  cost  drivers.  The 
struggle  between  centralized  and  decentralized  oversight  has  lasted  for  the  last  60  years, 
and  this  thesis  will  provide  insight  to  whether  the  streamlining  efforts  to  decentralize  the 
current  space  acquisition  procedures  are  being  accepted  by  those  using  the  new  NSSAP 
03-01  process. 

Even  prior  to  the  Air  Force’s  existence,  the  oversight  process  has  been  a 
constantly  fluctuating  process  that  moved  from  decentralization  to  centralization  of 
control  and  oversight.  This  chapter  will  introduce  the  background  of  the  problem  and 
show  how  this  research  is  needed  in  the  National  Security  Space  Acquisition 
environment.  This  chapter  will  provide  a  brief  history  of  current  knowledge  and  will  list 
all  assumptions  for  establishing  the  boundaries  for  the  research. 

1.2  Background 

Twenty-one  years  prior  to  the  creation  of  the  Air  Force,  the  oversight  of 
acquisition  programs  was  left  to  the  local  level  managers.  In  1947,  with  the  creation  of 
the  Air  Force,  the  first  Secretary  of  the  Air  Force,  Stuart  Symington,  established  a 
centralized  procurement  system  (5:9).  By  changing  the  location  of  oversight  approval 
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authority,  a  precedent  had  been  set  that  continues  to  be  inherent  in  today’s  acquisition 
oversight  process;  the  struggle  for  oversight  control.  For  the  next  56  years,  the  oversight 
process  changed  from  centralized  to  decentralized  and  back  again  10  times;  only  19 
percent  of  the  time  was  spent  decentralized.  The  most  recent  change  was  to  create  a  new 
National  Security  Space  Acquisition  Policy  that  gives  the  Director  of  the  National 
Reconnaissance  Office  (DNRO)/Under  Secretary  of  the  Air  Force  (USecAF)  the 
decentralized  control  to  make  the  key  decisions.  The  new  process  decentralizes  the 
decision  making  process  and  potentially  streamlines  the  approval  process  of  space 
programs  going  through  major  milestones  in  their  program  lifecycles.  This  step, 
however,  is  just  a  potential  round  in  the  fight  for  oversight.  To  place  the  struggle 
visually,  Figure  1 . 1  shows  the  length  of  time  the  acquisition  process  has  remained  in  the 
decentralization  model,  starting  in  1926.  When  the  process  is  decentralized,  the  points 
are  at  zero. 


Figure  1.1:  The  Struggle  for  Decentralization 
If  the  decentralization  of  government  actually  provides  cost  savings,  why  is  it  that 
the  maximum  time  spent  in  decentralization,  other  than  pre-World  War  II,  is  four  years? 
Does  loosening  centralized  control  make  it  easier  and  faster  to  procure  weapon  systems 
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and  does  that  outweigh  the  possible  increased  costs  of  fraud?  Is  there  a  perceived 
difference  in  the  actual  cost  of  perfonning  the  oversight  between  the  centralized,  or 
standard  DoDD  5000.1,  and  the  new  National  Security  Space  Acquisition  Policy? 

1.3  Problem 

This  research  will  answer  three  questions  and  provide  insight  to  the  answers  of 
those  questions.  There  are  three  questions  that  will  be  addressed  in  the  research. 

1 .  According  to  experts  in  the  field,  what  is  the  estimated  cost  of  oversight  for  a 
space  acquisition  program  under  the  NSSAP  03-01  IPA  oversight  process? 

2.  How  does  the  cost  of  oversight  for  space  programs  compare  to  the  cost  of 
oversight  for  programs  under  the  DoDD  5000  series  and  the  virtual 
acquisition  processes? 

3.  What  are  the  five  main  cost  drivers  for  the  oversight  of  space  acquisition 
programs? 

The  Key  Decision  Points  (KDP)  will  be  the  primary  measure  for  establishing 
costs  for  oversight.  These  are  the  points  in  the  acquisition  program  when  a  major 
decision  must  be  made  to  judge  if  a  space  system  can  move  forward.  For  each  KDP, 
experienced  people  whom  do  not  specifically  have  a  major  stake  in  the  program  being 
reviewed  must  perform  an  Integrated  Program  Assessment  (IPA).  If  the  IPA  finds, 
through  its  independent  review,  that  there  is  a  problem,  the  program  must  either  be  fixed 
prior  to  moving  forward,  or  can  move  forward,  with  approval  from  the  DNRO/USccAF, 
with  an  approved  plan  for  fixing  the  program.  The  IPA  process  occurs  a  minimum  of 
three  times  in  a  program’s  life-cycle. 

The  IPA  process  is  what  will  control  the  length  of  time  the  program  spends  in  the 
KDP.  The  process  takes  approximately  180  days,  not  including  the  initial  start  of  the 
Independent  Cost  Analysis/Estimate  (ICA/E).  Since  this  thesis  will  only  cover  the  IPA 
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process  from  day  10  forward  to  completion,  the  only  part  of  the  ICA/E  covered  are  the 
portions  included  in  the  standard  IPA  process. 

1.4  Summary  of  Current  Knowledge 

Knowing  that  the  process  of  decentralization  has  been  fought  for  over  56  years, 
the  main  focus  now  shifts  to  one  distinct  question:  exactly  why  is  the  establishment  of  the 
cost  of  the  new  oversight  process  so  important?  Over  the  past  56  years,  with  the 
revisions  of  the  acquisition  process,  whether  it  was  centralized  or  decentralized,  the 
revision  added  or  subtracted  points  in  the  oversight  that  cost  or  gained  the  program 
because  of  the  change  in  time  for  the  decision.  These  results  could  have  been  positive  or 
negative,  but  no  true  comparison  currently  exists.  For  the  DoDD  5000  series,  the 
oversight  process  is  a  series  of  decision  points  that  must  be  met  in  order  for  that  program 
to  progress  through  the  acquisition  lifecycle.  In  order  to  accomplish  this,  several 
meetings  with  various  stakeholders,  members,  and  principal  decision  makers  must  occur. 
There  is  normally  a  schedule  produced  with  estimated  completion  dates  set  for  each 
meeting.  In  1994,  the  Acquisition  Reform  Process  Action  Team  estimated  that  it  cost 
$10-12  million  dollars  in  oversight  for  a  program  to  pass  through  one  decision  point 
(17:9).  This  cost  was  taken  over  three  programs  and  the  exact  nature  of  the  programs 
could  not  be  ascertained.  Since  this  dollar  figure  is  nine  years  old,  the  figure  will  be 
adjusted  with  average  inflation  rates  in  Chapter  2  to  reflect  current  dollar  amounts.  This 
does  provide  a  comparative  baseline  when  completing  the  analysis  and  the  establishment 
of  the  IPA  cost  of  oversight. 

In  the  Report  of  the  Commission  to  Assess  United  States  National  Security  Space 
Management  and  Organization  (11),  the  Commission  focused  on  the  need  to  restructure 
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the  space  acquisition  process  through  the  reorganization  and  decentralization  of 
oversight.  The  main  reason  for  this  reorganization  is  the  differences  between  a  normal 
Major  Defense  Acquisition  Program  (MDAP)  and  a  Space  MDAP.  The  normal  system 
prepares  its  lifecycle  for  an  increased  cost  in  the  production  and  development  cycle  as 
well  as  the  operations  and  support  cycles. 
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Figure  1.2:  Milestone  Decision  Points  for  DoDD  5000.1  (33:14) 

Addressing  Figure  1.2,  the  points  of  interest  are  the  triangles  marked  A,  B,  and  C. 
At  these  points  are  the  major  milestones  when  key  decisions  are  made  and  the  process  of 
oversight  will  be  measured  and  compared.  When  this  process  is  compared  to  the  new 
NSSAP,  the  differences  caused  by  the  nature  of  the  acquisition  and  the  differences  in  the 
life  cycles  are  immediately  apparent. 
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A  review  of  the  literature  revealed  several  articles  on  the  revisions  and  the  history 
of  the  acquisition  process,  with  only  one  discovery  that  briefly  mentioned  the  actual  cost 
of  oversight,  the  Acquisition  Reform  Process  Action  Team  (17),  mentioned  earlier. 
Although  these  pieces  of  literature  provide  a  historical  perspective  and  allow  the  tracking 
of  the  oversight  process  changes  over  time,  it  is  the  goal  of  this  thesis  to  establish  the 
actual  cost  of  the  oversight  process  in  the  new  NSSAP  03-01. 

1.5  Assumptions 

Several  assumptions  are  relevant  for  the  present  research.  First,  one  KDP  will  be 
similar  to  the  next.  Since  the  process  takes  180  days,  only  one  process  will  be  reviewed. 
This  assumption  will  allow  for  an  easy  calculation  to  equate  to  the  entire  program  cost. 
The  next  assumption  is  that  the  oversight  process  will  be  limited  from  the  Program 
Executive  Officer  to  the  Director/NRO.  This  will  allow  for  a  traceable  medium  in  which 
the  time  the  Program  Manager  is  spending  in  meetings  for  the  IPA  will  be  gathered.  This 
will  also  allow  for  smooth  comparisons  with  the  other  forms  of  oversight  when  placed  for 
comparison.  Another  assumption  for  the  comparative  analysis  is  that  the  variances  are  all 
equal.  This  will  enable  a  simplistic  comparison  when  conducting  the  analysis  of  variance 
testing. 

1.6  Scope 

This  research  sets  out  to  find  the  cost  of  oversight  as  perceived  by  those  experts 
extensively  involved  in  the  space  acquisition  process.  The  estimates  will  be  obtained  by 
querying  a  panel  of  experts,  who  will  remain  anonymous,  with  questions  that  will 
ultimately  provide  a  group  consensus  on  the  cost  of  oversight  to  get  one  program  through 
one  KDP.  The  panel  of  experts  will  be  asked  to  estimate  from  the  PEO  to  the  final 
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decision  member,  which  is  the  USecAF,  the  cost  of  all  meetings,  support,  temporary  duty 
costs,  everything  involved  with  the  oversight  to  get  a  space  program  through  an  IPA  with 
an  approval.  The  costs  will  be  compared  to  a  report  filed  in  1994  by  the  Acquisition 
Reform  Process  Action  Team  (17),  the  DoDD  5000  series  research,  conducted  by  Captain 
Gary  Rousseau  (40);  and  the  C3I  RIT  (Virtual  Process)  research,  conducted  by  Captain 
Monroe  Neal  (32).  The  comparison  will  show  if  perception  of  the  streamlining  of  the 
acquisition  oversight  processes  have  been  accepted  fully  by  the  respective  acquisition 
panel  experts. 

The  process  will  not  include  anything  below  the  PEO  level  and  will  not  go  higher 
than  the  DNRO/USecAF.  This  will  allow  for  a  normalized  comparison  between  the  other 
programs  being  researched  by  Captain  Rousseau  and  Capt  Neal.  The  oversight  process 
will  be  reviewed.  This  research  will  not  include  external  influences  on  the  oversight 
process.  These  external  influences  are  the  budget,  the  requirements  and  generation 
process,  and  other  congressional/political  processes  that  inherently  affect  programs  and 
their  everyday  oversight  operations.  The  focus  of  this  research  is  to  establish  an  estimate 
of  the  cost  of  oversight  for  the  NSSAP  03-01.  The  goal  is  to  set  the  framework  and  the 
foundation  for  future  research  for  the  cost  of  oversight. 

The  research  will  stop  once  sufficient  data  has  been  collected  from  at  least  one 
KDP  going  through  the  IPA  process,  as  described  in  NSSAP  03-01.  Once  the  framework 
has  been  established,  the  research  will  conclude  with  the  analysis  of  the  results  and  the 
conclusion  of  the  findings.  The  possibility  of  future  research  on  this  topic  is  expected. 

1.7  Standards 
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Since  this  is  a  new  direction  in  the  research  on  oversight.  With  the  list  of 
assumptions  and  the  scope  of  the  research,  the  goal  will  always  be  maintained:  establish 
the  foundation  for  the  cost  of  oversight  for  the  NSSAP  03-01  acquisition  IPAT  KDP 
process.  The  comparison  with  the  1994  Process  Action  Team  will  be  necessary  to  equate 
their  cost  figures  to  the  cost  figures  gathered  in  this  research.  The  comparison  with  the 
research  conducted  by  Captain  Rousseau  and  Captain  Neal  will  provide  insight  in  testing 
for  differences  among  the  separate  acquisition  processes. 

1.8  Summary  of  Thesis 

The  collection  of  information  is  the  most  vital  portion  of  the  research.  After 
reviewing  the  proposal,  the  literature  review  will  provide  the  background  of  the  history  of 
oversight  and  the  research  conducted  on  the  acquisition  cycle.  The  third  chapter  will 
provide  the  methodology  for  gathering  data,  the  use  of  the  Delphi  method,  and  how  the 
research  focus  will  be  maintained.  The  fourth  chapter  will  display  the  results  of  the 
results  from  following  the  Delphi  methodology  and  will  lead  to  the  fifth  chapter  where  all 
analysis  will  be  completed  and  all  comparisons  will  be  made. 
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2.0  Literature  Review 


2.1  Overview 

As  of  31  December  2002,  there  are  72  categorized  major  defense  acquisition 
programs  (MDAPs)  in  the  Department  of  Defense  (DoD)  (20).  These  programs  currently 
total  over  $1.12  trillion  dollars  of  taxpayer’s  money  (20).  Of  these  programs,  seven  are 
considered  space  programs,  not  including  the  Ballistic  Missile  Defense  Program  (20). 
These  seven  programs,  as  of  December  2003,  totaled  $35.79  billion.  Although  these 
programs  constitute  only  three  percent  of  the  total  programs  listed  in  the  Selected 
Acquisition  Report  (SAR),  recent  changes  place  several  of  them  under  new  oversight 
guidance.  These  changes  will  alter  the  way  all  space  programs  are  managed  and  tracked 
in  the  future. 

A  literature  search  for  background  and  pertinent  information  revealed  limited 
research  on  the  actual  cost  of  oversight  for  MDAPs.  Several  documents  addressed 
contract  oversight,  but  only  one  touched  on  actual  cost  of  oversight  reviews.  The 
common  theme  found  in  the  research  uncovered  a  historical  chronological  struggle  with 
the  centralization  and  decentralization  of  oversight  on  the  acquisition  process. 

This  chapter  covers  topics  ranging  from  the  beginning  of  Air  Force  acquisitions, 
when  space  programs  began,  and  the  beginning  of  the  Department  of  Defense  Directive 
(DoDD)  5000  and  its  reforms.  It  also  contains  a  brief  history  of  space  acquisitions  and 
recognizing  the  need  for  change.  The  chapter  will  show  the  realigning  of  space 
acquisitions  and  seeing  if  the  new  acquisition  process  really  worked. 
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2.2  The  Beginning  of  the  Air  Force  Acquisition  Process 

Prior  to  World  War  II,  most  defense  acquisition  processes  were  handled  by  the 
Dayton,  Ohio  Air  Corps’  Materiel  Division.  “Although  some  key  decisions  were  made  in 
Washington,  DC,  the  Materiel  Division  played  a  critical  role  in  fostering  the  development 
of  American  aviation. . .”  (5:1).  At  that  time,  it  was  believed  that  most  of  the  oversight 
could  be  handled  locally  and  that  only  during  key  decision  times  should  Washington,  DC 
be  involved.  The  remainder  of  the  time,  local  leaders  were  expected  to  implement  the 
procedures  for  procuring  weapons  systems. 

In  1941,  the  Materiel  Division  split.  The  Army  created  the  Air  Corps 
Maintenance  Command  and  in  1942,  headquarters  moved  to  Patterson  Field  (5:7). 
Research,  Development,  Testing  &  Evaluation  (RDT&E)  became  Materiel  Command  in 
1942  and  moved  to  Wright  Field  in  1943  (5:7).  This  move  placed  acquisitions  and 
logistics  functions  in  practically  the  same  location. 

According  to  Benson,  the  move  in  1943  created  “confusion  and  duplication” 

(5:7).  In  1944  the  two  commands  were  merged  to  become  the  Army  Air  Force  Air 
Technical  Service  Command,  and  settled  at  Patterson  Field,  which  was  located  only  a 
few  blocks  away  from  Wright  Field.  At  this  time  “much  of  the  authority  over  acquisition 
matters  remained  in  Washington”  (5:7). 

With  the  end  of  World  War  II  and  the  creation  of  the  Air  Force  in  1947,  a 
centralized  procurement  system  was  planned  by  the  first  Secretary  of  the  Air  Force, 

Stuart  Symington.  The  goal  was  “to  perform  three  pair  of  core  functions:  (1)  research 
and  development,  (2)  supply  and  maintenance,  and  (3)  procurement  and  industrial 
planning”  (5:9).  This  decision  created  the  Air  Materiel  Command. 


11 


In  the  1950s,  the  Research  and  Development  Command  was  created  at  Wright- 
Patterson  Air  Force  Base,  Ohio,  which  had  merged  earlier  in  1948.  It  was  developed 
from  existing  infrastructure  and  missions.  Later  that  same  year,  it  was  renamed  the  “Air 
Research  and  Development  Command  (ARDC)...the  new  command  moved  its 
headquarters  to  Baltimore,  Maryland,  in  June  195 1”  (5: 10).  During  this  time,  space  was 
starting  to  be  explored  as  a  new  frontier  for  the  United  States.  A  new  acquisition  process 
had  to  be  initiated  in  order  to  maintain  the  security  of  the  United  States  in  the  new  space 
environment  and  ensure  that  the  weapon  systems  met  the  goals  of  the  United  States’ 
security  policies. 

2.3  Space  Acquisition  Programs  Begin 

From  the  beginning  of  space  exploration  in  the  1950s  to  today,  space  programs 
have  followed  the  standard  MDAP  acquisition  cycle.  Only  in  cases  of  secret,  or  “black 
world”,  operations  have  there  been  any  differences  between  the  acquisition  procedures  of 
the  two.  This  section  discusses  the  beginning  of  space  program  acquisitions  for  “open” 
programs  in  the  military.  The  “black  world”  operations  will  be  discussed  in  a  separate 
section. 

In  the  1950s,  a  new  organization  designated  the  Weapon  System  Program  Office 
(WSPO)  was  created.  “WSPOs  pulled  together  members  of  ARDC,  AMC,  and  (the) 
operational  command(s)  who  would  use  the  system. .  ..WSPOs  also  maintained  close 
liaison  with  the  contractors  involved”  (5:12).  The  use  of  the  word  “Weapon”  in  WSPO’s 
indicated  that  all  weapon  systems  would  be  included;  missiles,  aircraft,  and  new  space 
technologies.  This  weapons  systems  approach  was  based  on  a  new  acquisition  strategy 
called  “concurrency”  (5:12).  With  new  threats  of  nuclear  missiles  placed  in  countries 
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that  could  reach  the  United  States,  there  was  a  need  for  intennediate  range  and 
intercontinental  ballistic  missiles  (IRBMs  and  ICBMs)  and  to  acquire  them  as  quick  as 
possible.  The  concurrency  process  enveloped  all  of  the  previous  acquisition  steps  into 
overlapped  procedures  carried  out  in  a  coordinated  effort  that  was  “reminiscent  of  the 
streamlined  procurement  of  World  War  II”  (5:13).  Additional  problems,  in  this  case, 
included  the  need  for  the  new  technology  and  the  uncertainty  of  space. 

To  address  the  uncertainty  of  space  and  to  explore  the  space  frontier,  the  Air 
Force  Ballistic  Missile  Division  was  created  in  1957  (3 1:10).  The  Ballistic  Missile 
Division  managed  programs  including  the  Thor  IRBM  and  the  WS-1 17L  reconnaissance 
satellite  program.  The  Ballistic  Missile  Division  had  the  luxury  of  funding  priority  after 
the  launch  of  the  Soviet  satellite  “Sputnik”  in  1957  (5:13).  With  this  funding  priority,  the 
Ballistic  Missile  Division  developed  and  deployed  missiles  and  subsystems,  built  launch 
sites  and  support  equipment,  and  trained  crews;  all  simultaneously.  “Although  this 
effort... led  to  cost  overruns,  extensive  modifications,  and  unrealistic  training,  it  truly 
achieved  the  goal  of  giving  the  United  States  a  ballistic  missile  deterrent  as  soon  as 
possible”  (5:14). 

In  1951,  the  test  and  evaluation  portion  of  the  acquisition  process  had  split  into 
seven  phases;  in  1956,  it  increased  to  eight  (5:14).  The  Air  Force  recognized  the  need  to 
cut  the  costs  included  with  the  extensive  oversight  process.  In  1957,  to  “help  expedite 
production  decisions”  (5:14),  the  Air  Proving  Command  was  abolished  and  in  1958,  the 
eight-phase  test  portion  was  changed  to  a  three-phase  system. 

In  1958,  the  National  Aeronautics  and  Space  Administration  (NASA)  was 
activated  as  a  non-DoD  agency  for  the  peaceful  exploration  of  space  by  the  United  States 
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(27:1 1).  With  this  new  agency,  the  military  was  allowed  to  play  an  important  support 
role  in  scientific  programs  and  offered  several  assets  to  the  new  agency. 

In  1959,  “Headquarters  Air  Force  formed  the  Weapon  Systems  Management 
Study  Group”  (5:15).  The  goal  was  to  review  the  current  acquisition  cycles  and  the  issue 
of  concurrency.  The  group  reached  compromises  that  resulted  in  the  “better  defining 
authority  at  each  stage  of  the  acquisition  strategy”  (5:15).  The  group  created  a  new  series 
of  acquisition  regulations  and  renamed  the  WSPO  the  System  Program  Office  (SPO). 
This  movement  “recognized  the  growing  importance  of  C3,  surveillance,  and  other 
technologies  that  supported  the  war-fighter  (5:15). 

In  1961,  there  was  “an  offer  the  Air  Force  couldn’t  refuse,”  the  control  and 
responsibility  for  military  space  programs.  This  control  came  with  the  condition  that  the 
current  acquisition  process  had  to  change  to  better  handle  the  unique  mission  of  space. 

By  1963,  however,  the  new  oversight  in  the  acquisition  process  had  become  burdensome. 
“At  the  top  was  the  Office  of  the  Secretary  of  Defense  (OSD)  and  its  new  requirements 
for  cost  effectiveness  data  and  disciplined  programming  and  budgeting  schedules”  (5:16). 
The  oversight  seemed  to  trickle  down  the  process  until  it  reached  the  contractor.  The 
new  processes  made  programming  and  budgeting  difficult  and  the  new  scope  of  the 
programs  “made  cost  predictions  difficult  and  led  to  unrealistic  bids”  (5:16).  Another 
problem  was  the  increased  number  of  reports,  audits,  proposals,  studies  and  the  other 
oversight  items  requested  by  OSD. 

In  1969,  after  several  cost  overruns  and  under  Congressional  pressure,  President 
Nixon  commissioned  a  round  of  acquisition  reforms.  Then  Deputy  Secretary  of  Defense, 
David  Packard,  was  chosen  to  lead  these  efforts.  The  result  would  lead  to  many  years  of 
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oversight  battle  and  reforms  of  the  acquisition  process.  A  closer  look  at  the  reforms  and 
the  resulting  documents  follow  in  the  chain  of  events. 

2.4  The  Beginning  of  DoDD  5000  and  Its  Reforms 
2.4.1  The  Start  of  the  DoDD  5000  series 

During  the  1969  review,  Secretary  Packard  established  the  Defense  Systems 
Acquisition  Review  Council  (DSARC),  which  essentially  served  the  same  functions  as 
today’s  Defense  Acquisition  Board  (DAB)  or  Defense  Space  Acquisition  Board  (DSAB) 
(1:74).  The  DSARC’s  main  function  was  to  advise  the  Secretary  of  Defense  of  anything 
considered  significant  in  the  acquisition  of  a  defense  program.  Secretary  Packard  also 
asked  that  the  Director  of  Defense  Research  &  Engineering  conduct  management  reviews 
“at  least  once  on  each  major  acquisition  program”  (1:74). 

Secretary  Packard  had  other  ideas  for  improving  the  acquisition  process.  He 
published  these  ideas  in  a  memorandum  that  became  the  foundation  for  the  first  DoD 
Directive  5000.1  published  in  1971.  The  new  “5000  series”  (1:75)  called  for 
“decentralization  of  responsibility  and  authority  for  the  acquisition  of  major  defense 
systems  to  the  greatest  extent  possible,  consistent  with  the  urgency  and  importance  of  a 
particular  defense  system  being  acquired”  (5:75).  Secretary  Packard  recognized  the  need 
to  limit  oversight  of  programs  to  only  items  that  could  potentially  damage  the  system. 

His  basic  premise  was  that  if  anything  was  wrong  with  the  program,  it  would  be 
discovered  at  the  lowest  level.  Therefore,  DoDD  5000. 1  “ordered  that  program  managers 
be  given  adequate  authority  to  make  major  decisions,  rewards  for  good  work,  and  more 
recognition  toward  career  advancement”  (5:75).  With  the  new  DoDD  5000  series,  the 
DoD  assumed  “responsibility  for  establishing  acquisition  policy....”  (5:75),  but  the  DoD 
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services  were  responsible  for  identifying  their  needs  and  actually  acquiring  the  items  to 
fulfill  them.  The  first  of  the  5000  series  was  only  seven  pages  long  and  stated  the  duties 
of  only  three  DoD  officials  (21:111). 

2.4.2  Past  Reforms  to  DoDD  5000.1 

Ferrara  (21:113)  points  out  that  “with  very  few  exceptions  there  has  not  been 
wide  variation  in  the  fundamental  management  principles  underlying  the  defense 
acquisition  system.”  Even  the  first  published  version  of  DoDD  5000. 1  pushed  for  the 
centralization  of  policy  making,  but  emphasized  decentralized  program  execution. 
Ferrara  cites  two  statements  made  in  two  different  versions  of  the  DoDD  5000  series. 
The  first  is  from  the  1971  version: 

Responsibility  and  authority  for  the  acquisition  of  major  defense  systems  shall  be 

decentralized  to  the  maximum  extent  practicable  consistent  with  the  urgency  and 

importance  of  each  program  (21:113). 

The  1977  version  states: 

Responsibility  for  the  management  of  system  acquisition  programs  shall  be 
decentralized  to  the  DoD  Components  except  for  decisions  retained  by  the  Secretary  of 
Defense  (21:113). 

This  oversight  decentralization  recommendation  goes  back  to  the  acquisition 
policies  in  use  the  years  prior  to  World  War  II  when  the  key  program  decisions  were 
made  in  Washington  and  the  routine,  system  program  decisions  were  made  by  the  ones 
performing  the  day-to-day  tasks  of  running  the  program. 

With  each  new  Presidential  tenn,  there  has  been  at  least  one  change  to  the  DoDD 
5000  series.  The  revision  in  1975  created  DoDI  5000.2,  the  “instruction  guidelines 
governing  the  use  of  the  Decision  Coordinating  Paper  (DCP)  and  the  DSARC”  (21:116). 
With  the  new  DoDI  5000.2,  the  DSARC  was  only  an  advisory  panel  to  the  Secretary  of 
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Defense  on  MDAPs.  This  format  was  held  in  place  for  two  years,  until  1977  and  the 
development  of  a  new  milestone  was  implemented.  The  1977  revision,  signed  two  days 
prior  to  President  Carter’s  inauguration,  put  an  end  to  DoDI  5000.2  and  made  it  an  actual 
DoD  Directive  (21:117).  The  1977  revision  cancelled  the  DSARC  and  placed  the  duties 
and  responsibilities  of  the  DSARC  members  directly  into  the  directive.  “The  major 
change  evident  in  this  version  was  the  addition  of  a  new  milestone. . .  .demonstration  and 
validation”  (21:117).  The  two  prior  versions  of  DoDD  5000  contained  only  three 
milestones. 

In  1977,  after  all  these  additional  oversight  additions,  a  comment  was  made  by  the 
Chairman  of  the  Acquisition  Cycle  Task  Force  of  the  Defense  Science  Board: 

“These  procedural  changes  have  become  institutionalized  and  have  been  applied 
inflexibly  to  all  programs  with  the  result  that  the  acquisition  process  has  steadily 
lengthened  and  the  procurement  of  defense  systems  has  become  increasingly  costly” 
(1:79).  This  statement  synopsizes  a  perception  of  constant  change  and  refonn  that 
perhaps  shows  no  signs  of  stopping. 

The  1980  revision  of  DoDD  5000,  again  added  a  new  document  to  “summarize 
the  implementation  plan  of  the  DoD  Component  for  the  life  cycle  of  the  system” 

(21:118).  This  revision  added  more  oversight  and  more  documentation  to  the  acquisition 
process.  This  swing  of  centralization  and  decentralization  of  oversight  continues  to 
fluctuate  throughout  the  history  of  the  DoDD  5000  series.  Even  in  1982,  with  the 
implementation  of  the  Carlucci  Initiatives,  the  decision  to  centralize  the  oversight  process 
was  tied  to  DoDD  5000.  The  Carlucci  Initiatives  contained  32  management  initiatives, 
including  multiyear  procurement  and  linking  acquisition  and  budgeting.  Some  of  these 
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ideas  presented  were  reminiscent  of  the  old  “concurrency”  process  from  the  beginning  of 
the  space  acquisition  programs  and  in  the  years  prior  to  World  War  II.  By  1982,  the  goal 
was  to  reduce  oversight  and  increase  responsibility  at  the  lowest  level.  With  the  media 
reports  of  $500  hammers  and  $300  toilet  seats  in  the  mid-1980s,  Congress  demanded  new 
acquisition  oversight  procedures.  There  came  a  point  when  “the  logjam  of  procurement 
legislation  awaiting  implementation  had  become  so  great  that  the  Pentagon  and  defense 
industry  officials  pleaded  with  Congress  for  a  moratorium  on  further  reform  legislation” 
(21:119).  The  biggest  change  in  that  time  period  was  the  creation  of  the  Defense 
Acquisition  Executive  (DAE).  This  new  position  was  established  to  show  Congress  that 
the  Pentagon  was  taking  steps  in  the  right  direction  to  fix  its  acquisition  problems.  This 
assurance  was  again  accomplished  by  increasing  oversight  and  adding  layers  to  an 
already  burdensome  system. 

In  1985,  Packard  was  requested  by  President  Reagan  to  fonn  another  Commission 
on  acquisition.  In  1987,  this  commission  attempted  to  issue  new  recommendations  based 
on  Packard’s  original  ideas  when  the  first  DoDD  5000  was  issued.  One  new 
recommendation  was  the  creation  of  an  Under  Secretary  of  Defense  for  Acquisition 
(USD(A)).  Even  though  the  newest  revision  was  deemed  controversial,  it  made 
significant  changes  that  are  still  seen  today.  One  of  those  major  changes  was  the  creation 
of  the  Defense  Acquisition  Board  (DAB).  This  group  was  slightly  larger  than  its 
predecessor,  the  DSARC,  but  it  essentially  served  the  same  purpose.  DABs  were  created 
for  each  specific  section  (i.e.,  aircraft).  Each  DAB  was  to  become  the  “experts”  in  that 
area.  When  initially  started,  there  were  a  total  of  10  DABs,  some  focusing  on  specific 
topics,  but  most  focused  on  broad  issues.  At  one  point,  there  were  at  least  126  different 
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DABs  and  committees  that  reported  to  the  new  USD(A)  (21 : 120).  Another  item  with  the 
1987  revision  was  the  increasing  of  milestones  from  four  to  six.  These  milestones  added 
oversight  to  an  area  where  it  was  felt  that  oversight  was  needed,  at  the  beginning  and  the 
middle  of  the  production  cycle.  In  the  attempt  to  streamline  the  acquisition  process,  the 
Packard  Commission  inadvertently  added  additional  layers  to  the  process. 

In  1991,  oversight  became  more  centralized  only  to  become  decentralized  again  in 
a  revision  in  1996.  The  new  1991  revision  made  the  DoDD  5000.2  an  instruction  once 
more,  but  added  a  new  DoD  5000.2-M  Manual  to  the  stack  of  documents.  The  DoDD 
5000  series,  as  a  basic  “cookie  cutter”  acquisition  procedure  for  MDAPs,  was  now 
creating  a  policy  of  “no  supplements”  for  this  version  and  that  it  was  for  all  programs, 
regardless  of  size  (21:121).  The  1991  version  ballooned  to  approximately  900  pages  and 
152  references  to  other  sources.  The  1991  documents  also  created  acquisition  categories, 
or  “ACATs”  (2 1 : 122).  Although  the  statutory  requirements  of  Title  10  were  still  upheld, 
it  created  additional  tiers  of  oversight  for  the  smaller  acquisition  programs  due  to  the 
newly  created  AC  AT  levels. 

After  this  new  oversight  was  created,  the  1996  version  of  DoDD  5000  was  issued 
and  removed  some  of  the  past  restrictions  and  centralization.  The  1996  version  came 
with  four  objectives  (21:122-123): 

1 .  Clearly  separate  mandatory  policies  and  procedures  from  discretionary 
practices.  This  decentralized  control  and  allowed  managers  to  once  again 
attempt  to  control  their  programs. 

2.  Took  new  laws  and  regulations  since  the  last  update  and  included  them  in  the 
series. 

3 .  Consolidated,  for  the  first  time,  policies  for  MDAPS  and  automated 
infonnation  systems. 

4.  Finally,  try  and  stop  the  overall  feeling  that  the  5000  series  was  getting  too 
large  and  too  complex  to  understand  and  implement. 
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The  goal  was  to  make  it  easier  to  use  and  provide  understandable  guidance  for  all 
major  systems.  The  latest  revision  included  space  programs.  The  1996  version  also 
implemented  a  recommendation  from  the  Acquisition  Refonn  Process  Action  Team  of 
having  Integrated  Product  Teams  (IPTs)  for  the  acquisition  programs  (17:38). 

Since  1996,  there  have  been  two  additional  revisions  to  the  DoDD  5000  series, 
one  of  them  being  finalized  at  the  time  of  this  research  effort.  The  new  DoDD  5000 
series  will  implement  more  decentralization  than  previous  editions.  One  of  the  major 
changes  in  the  acquisition  process  is  the  ability  to  separate  acquisition  programs  into 
different  acquisition  procurement  channels.  This  separation  is  accomplished  by  allowing 
waivers  from  the  constraints  of  DoDD  5000  and  places  the  program  into  these  separate 
proposed  tracks.  These  proposed  tracks  are  currently  acquisition  under  DoDD  5000 
series,  space  systems  acquisitions,  Ballistic  Missile  Defense  acquisitions,  and  acquisitions 
operating  in  a  virtual  realm  which  is  called  C3I  RIT.  All  of  these  processes  have  some 
connection  to  the  basic  DoDD  5000  series,  but  each  has  its  own  perceived  cost  saving 
methodology.  Because  this  research  concentrates  on  space  systems,  it  is  the  acquisition 
reform  process  as  it  applies  specifically  to  the  history  of  space  acquisition  that  needs 
particular  attention.  Space  acquisition  had  its  beginning  in  the  “black  world.”  In  order  to 
relate  the  transformation  of  oversight  to  the  new  space  acquisition  process  the  history  of 
the  Corona  Project  is  reviewed. 

2.5  History  of  Space  Acquisition-the  Corona  Project 

The  United  States  space  acquisition  program  was  started  almost  by  accident. 

Space  was  barely  mentioned  or  considered  until  the  United  States  captured  and 
interviewed  the  creator  of  the  V-2  rocket,  German  scientist,  Dr.  Werner  von  Braun,  who 


20 


submitted  a  report  about  the  future  of  space  and  “included  ideas  about  satellites”  (34:4). 
This  report  caught  the  Navy’s  attention  and  the  department  was  interested  in  creating  a 
satellite  test  program. 

By  the  time  two  different  test  studies  were  completed,  the  United  States  Air  Force 
had  been  created  and  in  1948,  Gen  Vandenberg,  Air  Force  Chief  of  Staff,  learned  that  the 
Air  Force  “had  the  logical  responsibility  for  the  satellite”  (34:6).  But  even  after  the  Navy 
withdrew  its  request  for  its  own  satellite  programs,  the  remaining  program  was  given 
limited  funding  and  restricted  on  the  research  and  development  of  the  satellite. 

In  the  early  1950s,  the  RAND  (Research  ANd  Development)  company  was 
awarded  a  contract  to  do  a  feasibility  study  on  the  use  of  satellites  for  photographic 
imagery  of  earth.  During  RAND’s  study,  another  group  was  establishing  the 
requirements  for  reconnaissance  for  the  years  1952  through  1960.  The  concept  of 
satellite  use  was  never  entertained  in  the  reconnaissance  plan.  The  only  comment  made 
was  that  satellite  systems  would  not  be  worth  the  financial  outlay  required. 

In  September  1953,  the  RAND  report  was  sent  to  the  ARDC  for  possible 
development  and  design.  By  December  of  that  same  year,  “Project  409-40”  was  bom.  It 
was  given  the  unofficial  designation  of  “Weapon  System  1 17L”  (34: 15).  Since  the 
design  exploration  was  completed,  the  next  step  was  to  find  a  contractor  who  would 
attempt  to  put  the  WS-1 17L  into  space.  In  June  1956,  Lockheed  was  awarded  the 
contract  (34:25).  The  biggest  challenge  the  program  faced  was  the  limited  budget  of  $3 
million  for  a  program  that  required  extensive  testing  of  the  design  and  the  technology 
needed  had  not  been  developed.  In  November  of  that  same  year,  the  program  was  halted 
until  the  launch  of  the  Russian  satellite,  Sputnik,  had  shown  the  need  for  satellites.  After 
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the  Sputnik  launch  in  1957,  the  need  for  the  United  States  to  get  reconnaissance  into 
space  became  a  top  priority.  When  the  true  satellite  concept  was  finally  realized,  it  was 
set  up  to  be  a  generalized  program  with  minimal  oversight.  The  basic  program  was  run 
with  the  basis  of  “instructions  were  not  so  much  marching  orders  as  ‘marching 
suggestions’”  (34:43). 

With  the  limited  oversight  established  under  the  Corona  Project,  from  the  initial 
concept  exploration  in  1953  to  the  first  launch  in  1959,  showed  how  quickly  a  program 
can  move  with  decentralized  oversight.  The  process  was  seen  as  successful  because  “the 
group  was  so  small  and  its  members  were  so  close  that  decisions  could  be  made  jointly” 
(34:43).  From  the  beginning  of  the  launches  in  1959  until  the  program  ended  in  1972, 

145  satellites  were  developed,  enhanced,  improved,  and  launched. 

The  Corona  project  was  conducted  in  a  shroud  of  secrecy  that  did  not  follow  the 
normal  acquisition  guidelines  of  those  times.  That  theory  carried  forward  as  the  National 
Reconnaissance  Office  (NRO)  was  bom  on  10  June  1960  (34:94).  Even  through  the 
1990s,  a  separate  acquisition  process  was  used  by  the  NRO,  separate  from  other  agencies. 
This  process  contained  limited  oversight  and  a  perception  of  quick  acquisition  timelines. 
However,  as  space  became  more  available  to  the  world,  the  use  of  satellites  and  missiles 
became  more  open.  With  the  creation  of  the  DoDD  5000  series,  more  of  the  space 
acquisitions  fell  under  the  control  of  the  directives  and  was  placed  under  the  standard 
oversight  process  for  acquisitions.  This  gradual  move  brought  the  once  decentralized 
process  under  a  centralized  control. 

With  the  knowledge  of  the  NRO  acquisition  process  and  the  foresight  of  the  need 
for  acquisition  oversight  reforms,  the  DoDD  5000  acquisition  guidance  was  seen  as 
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ineffective  for  space  programs.  A  change  was  needed  and  a  new  direction  was  taking 
place  in  space  acquisition. 

2.6  Recognizing  the  Need  for  Change 

On  1 1  July  2001,  a  report  was  published  by  the  Commission  to  Assess  United 
States  National  Security  Space  Management  and  Organization.  In  this  report,  several 
aspects  of  the  United  States  Space  Program  were  reviewed.  One  key  point  was  the  way 
the  acquisition  environment  and  oversight  of  space  was  handled  by  different 
organizations  and  the  need  for  central  control.  One  item  for  potential  change  was  the 
way  the  infonnation  was  presented  to  key  decision  makers  and  another  was  who  those 
key  decision  makers  were.  As  seen  in  Figure  2.1,  the  flow  of  information  is  sent  from 
different  locations  for  decisions  for  different  levels.  This  type  of  oversight 
“responsibility  and  accountability  for  space  are  broadly  diffused  throughout  the 
government”  (11  :xix). 
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Figure  2.1:  Acquisition  Management  Chain  of  Command  (1  l:xix) 

In  the  Commission  on  Space  report,  it  is  shown  that  both  the  military  services  as 
well  as  the  NRO  execute  specific  space  programs.  The  military  leaves  it  up  to  each 
service  to  define  and  implement  their  own  space  program  to  meet  their  needs,  but  “no 
single  service  has  been  assigned  statutory  responsibility  to  ‘organize,  train  and  equip’  for 
space  operations”  (11  :xxii).  According  to  the  report,  “eighty- five  percent  of  space- 
related  budget  activity  within  the  (DoD). . .  .resides  with  the  Air  Force”  (11  :xxii). 

The  NRO  currently  “is  the  single  national  organization  tasked  to  meet  the  U.S. 
Government’s  intelligence  needs  for  space-home  reconnaissance”  (11  :xxiii).  The  NRO 
is  a  joint  venture  between  the  DoD  and  the  CIA.  With  their  concerted  effort,  they  try  to 
maintain  the  needs  of  the  intelligence  community. 
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Highlighting  the  differences  between  the  NRO  acquisition  process  and  the  main 
DoD  process  followed  by  the  Air  Force,  the  Commission  shows  that  the  NRO  takes  a 
“cradle-to-grave”  approach  while  the  Air  Force  separates  the  process  among  different 
commands  (1  l:xxvi).  Understanding  these  differences,  a  chance  to  merge  the  above 
items  to  create  a  synergistic  effort  for  space  acquisitions  programs  is  needed.  With  the 
Space  Commission’s  recommendation,  a  new  organizational  structure  was  proposed. 


Figure  2.2:  Space  Commission’s  Proposed  Space  Acquisition  Organization  (11  :xxvi) 
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Under  the  proposed  organizational  structure,  some  highlights  include; 

1.  Reorganizing  the  Air  Force  and  making  it  the  sole  responsibility  for  MDAPs  in 
the  Space  environment. 

2.  Align  the  Air  Force  and  the  NRO  Space  Programs.  The  Commission  to  Assess 
Space  proposed  to  “Assign  the  Under  Secretary  of  the  Air  Force  as  the 
Director  of  the  National  Reconnaissance  Office.  Designate  the  Under 
Secretary  as  the  Air  Force  Acquisition  Executive  for  Space”  (1  l:xxxiv). 

As  in  1961,  when  the  Air  Force  was  tasked  with  the  responsibility  to  take  over  the 
oversight  of  space  acquisition  programs,  the  plan  was  again  set  in  motion  in  2001.  With 
the  appointment  of  the  Chairman  of  the  Commission  to  Assess  U.S.  National  Security 
Space  Management  and  Oversight  to  his  new  position  as  the  Secretary  of  Defense,  the 
Honorable  Donald  Rumsfeld  was  able  to  implement  the  ideas  that  the  Commission  had 
created  as  he  became  the  new  Secretary  of  Defense. 

2.7  The  Realigning  of  Space  Oversight 

With  the  findings  of  the  Commission  to  Assess  Space  and  a  new  Secretary  of 
Defense  in  charge,  on  18  October  2001,  Secretary  Rumsfeld  signed  a  memorandum  that 
stated,  “I  agree  with  the  Commission’s  conclusion  that  a  new  and  comprehensive 
approach  to  national  security  space  management  and  organization  is  needed. ...”  (41 : 1). 
With  this  letter,  the  implementation  of  the  new  national  security  space  program  was  set  in 
motion.  Secretary  Rumsfeld’s  letter  was  preceded  by  a  press  conference,  given  by  him, 
highlighting  the  changes.  In  that  conference,  he  stated,  “The  majority  of  these  changes 
involve  realigning  Air  Force  headquarters  and  field  commands  to  more  effectively 
organize,  train  and  equip  for  space  operations,  ensuring  that  the  Air  Force  will  become 
the  lead  for  space  activities  in  the  Department  of  Defense”  (45:2). 

The  first  recommendation  was  to  establish  a  policy  for  acquisition  that  could 
easily  be  followed,  but  maintained  the  goals  of  the  Commission.  With  the  proposal  to 
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emulate  the  best  practices  of  all  services  and  the  basic  format  of  the  NRO  acquisition 
process,  NRO  Directive  7  was  used  as  the  foundation  for  the  new  space  acquisition 
policy.  In  Directive  7,  the  general  policy  is,  “the  NRO  will  acquire  systems  using  a  fast- 
paced,  streamlined  management  process”  (3 1  :Attach  1 :2).  The  policy’s  objective  is  to 
create  frequent  interaction  between  the  decision  makers  and  the  individual  program 
managers.  The  NRO’s  Directive  focuses  on  decentralized  decision  making  except  for 
three  key  periods  in  the  program’s  life-cycle,  which  are  called  “key  decision  points”  or 
KDPs  (3 1 :  Attach  1 :2).  The  NRO  follows  a  program  where  all  data  is  collected  prior  to 
the  KDP  for  each  milestone  and  a  team  of  experts  is  collected  for  the  purpose  of 
evaluating  the  program  independently,  and  providing  a  status  for  the  Director,  NRO. 

This  team  is  known  as  the  Integrated  Program  Assessment  Team  (IP AT)  and  the  process 
is  called  the  Integrated  Program  Assessment  (IP A)  (3 1  :Attach  1 :5).  To  complete  the 
KDP,  the  IP  AT  must  give  the  program  a  passing  evaluation  for  the  NRO  Acquisition 
Board  (NAB),  where  the  NRO  Director  makes  the  decision  to  move  forward.  This 
process  is  perceived  as  successful  due  to  the  limited  oversight  and  the  fact  that  the  IPA 
members  have  no  stake  in  the  outcome  and  provide  a  true  independent  review. 

Taking  the  best  practices  from  the  other  services  and  emulating  the  NRO  process, 
the  new  space  system  acquisition  policy  was  established.  With  a  series  of 
memorandums,  delegations  were  made  to  comply  with  Secretary  of  Defense’s  vision  and 
to  comply  with  the  recommendations  of  the  Space  Commission.  Table  2.1  lists  the  chain 
of  these  memorandums  and  the  direction  in  which  they  moved  the  space  acquisition 
system. 
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Table  2.1:  Policy  Letters  Establishing  Space  Acquisition  Oversight 


Date 

Subject 

From 

Jo 

Reason 

No  Date 
Given 

AF  Senior  Procurement 
Executive  Authorities 
and  Responsibilities 

SECAF 

USECAF 

Designation  of  USECAF  to 
Senior  Procurement 
Exectutive  (SPE)  (37) 

No  Date 
Given 

AF  Senior  Procurement 
Executive  Authorities 
and  Responsibilities 

USECAF 

Assistant  Secretary  of  the 
Air  Force  (Acquisition) 
ASAF(A) 

Designation  of  ASAF(A)  to 
Senior  Procurement 
Exectutive  (SPE)  (46) 

4  Feb  2002 

Delegation  of  Milestone 
Decision  Authority  for 
DoD  Space  Systems 

USECDEF 

All  Military  Secretaries 
Under  Secretary  of 
Defense(Comptroller) 
Assistant  Secretary  of 
Defense  (C3I) 
USECAF/DNRO 

Notification  of  Space 
program  MDAPs  MDA  to 
SECAF,  but  re-delegation 
limited  to  USECAF/DNRO 
(2) 

7  Feb  2002 

AF  Acquisition 
Executive  Authorities 
and  Responsibilities 

SECAF 

USECAF 

ASAF(A) 

Assisting  in  the  fulfillment  of 
Air  Force  Acquisition 
Executive  duties  (36) 

14  Mar  2002 

Re-Delegation  of 
Milestone  Decision 
Authority  for  DoD 
Space  Systems 

SECAF 

USECAF/DNRO 

Re-delegated  all  MDA  to 
USECAF/DNRO 
(39) 

1  Apr  2002 

Space  Based  Radar 
Acquisition 

USECAF 

USECDEF  (AT&L) 

Established  SBR  as  a  test 
program  for  the  new  space 
acquisition  process  (51) 

12  Apr  2002 

Organizational  Stand- 
Up  of  Executive  Agent 
for  Space 

DoD 

SECAF 

administrative 

assistant 

Distribution  C 

Eliminated  and  created 
specific  chains  for  the  space 
acquisition  process  (14) 

29  May  2002 

Fiscal  Year  (FY)  2002 
Major  Defense 
Acquisition  Program 
(MDAP)  Lists 

USECDEF 

All  Secretaries  of  the 
Military  Departments  and 
Director,  Missile  Defense 
Agency 

Redesignated  some  space 
programs  for  lower  levels  of 
oversight  IAW  the  new  space 
process (3) 

2  Oct  2002 

Independent  Cost 
Estimates 

USECDEF 

All  Secretaries  of  the 
Military  Departments 
USECDEF(Comptroller) 
ASECDEF(C3I) 
USECAF/DNRO 

Eliminated  the  need  for  CAIG 
as  responsible  for  Cost 
Estimate — Established  that 
the  AF  has  sole  responsibility 
to  comply 
(4) 

21  Nov  2002 

Defense  Space 
Acquisition  Board 

USECAF 

Several  Recipients 
(Too  long  to  list) 

Establishes  the  Director  of 
National  Security  Space 
Integration  as  the  Executive 
Secretary  for  the  Defense 
Space  Acquisition  Board 
(47) 

4  Mar  2003 

National  Security  Space 
(NSS)  Acquisition 

USECAF 

All  Service  Secretaries, 
CJCS 

USECDEF  (AT&L), 
ASECDEF(C3I), 
Director, (OT&E),  DDNRO 

Authorizes  the  release  of  NSS 
Acquisiton  Policy  03-01  as 
interim  guidance 
(50) 

20  Mar  2003 

Exemption  and  Waiver 
to  DoDI  5000.2  and 
Related  Guidance  for 

AF  Program  Executive 
Officer  for  Space 
(AFPEO/SP)  Space 
Programs 

USECAF 

Air  Force  Program 
Executive  Office  for  Space 

Grants  exemption  and 
waivers  from  DoDI  5000.2 
for  use  of  streamlined  space 
acquisition  process 
(48) 

29  May  2003 

Full  Funding  Policy  for 
DoD  Space  Programs 

USECAF 

D.PA&E 

Recommends  the  elimination 
of  “full  funding  at  KDP” 
policy  (49) 

7  Jul  2003 

Delegation  of  (DoD) 
Executive  Agent  for 
Space  Responsibilities 

SECAF 

USECAF 

Delegates  all  DoD  Executive 
Agent  for  Space  to  USECAF 
(38) 
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With  the  memorandums  channeled  and  the  approval  received,  the  National 
Security  Space  Acquisition  Policy  (NSSAP)  03-01  was  implemented  as  interim  guidance. 
The  NSSAP  03-01  incorporates  much  of  NRO  Directive  7,  along  with  a  few  best 
practices.  The  goal  was  to  tailor  the  acquisition  process  for  space  programs  so  the  needs 
of  National  Security  could  be  met  in  a  streamlined  process  with  a  limited  and 
decentralized  oversight  process.  One  reason  noted  in  a  Government  Accounting  Office 
(GAO)  report  focused  on  common  problems  in  acquisition  in  military  space  programs.  It 
was  noted  that  the  life-cycle  spending  on  the  space  programs  differs  from  spending  on 
aircraft  MDAPs.  Normal  DoD  lifecycle  profiles  show  that  28  percent  of  a  normal 
program’s  budget  is  spent  in  its  development,  while  72  percent  goes  into  the  operations 
and  support  items  for  a  certain  period  of  time,  as  seen  below. 


Figure  2.3:  Funds  Allocation  in  MDAP  (33) 

For  the  space  environment,  the  budget  expenditure  profile  is  exactly  the  opposite. 
The  funds  are  needed  up  front  to  ensure  the  system  will  work  once  launched  (23:8).  For 
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space  programs,  a  traditional  full  rate  production  effort  does  not  exist  like  there  is  in 
aircraft  operations  and  support.  On  space  programs,  much  of  the  work  is  done  up  front  to 
eliminate  any  potential  system  failures  once  the  satellite  is  launched.  There  are  very  few 
second  chances  once  a  satellite  system  is  placed  in  service.  The  true  expenditure  profile 
of  a  space  life  cycle  cost  curve  can  be  seen  in  Figure  2.4  in  comparison  to  a  typical  DoD 
life  cycle  cost  curve. 


Typical  DoD  Life  Cycle  Cost  Curve 

Notional  Space  Life  Cvcle  Cost  Curve 

Operations  &  Support 


System  Acquisitf 

_ A. 


System  Acquisition 

A 


Approx 
70% 


m 

KDP  KDP  KDP 


Operations  &  Support 

A_ 


Notional  Life  Cycle  Curve 


Figure  2.4:  Comparing  Normal  MDAP  to  Space  MDAP  (33) 

Because  of  this  new  process,  new  oversight  controls  are  implemented  as  well. 
With  NSSAP  03-01,  the  IPA  process  is  brought  up  in  full  force  with  full  explanations  of 
who  is  responsible  for  specific  duties.  Once  the  Program  Manager  feels  he/she  is  ready 
for  a  DSAB,  a  request  is  made  and  a  date  is  set,  and  the  calendar  is  rolled  back  for 
specific  points  in  time  to  be  met.  A  graphical  presentation  is  given  in  NSSAP  03-01  and 
is  replicated  for  easy  interpretation  of  the  IPA  process.  The  IPA  process  is  repeated  once 
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the  program  has  reached  a  KDP  for  each  KDP  in  the  satellite’s  life-cycle  acquisition 
profile. 
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Figure  2.5:  Extract  from  NSSAP  03-01  on  IPA  Process  (19) 
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With  the  new  process  and  information  flow  created  by  NS  SAP  03-01,  oversight  is 
expected  to  decentralize  and  is  hypothesized  by  some  as  being  a  cost  effective  way  to 
manage  MDAPs  across  all  services. 

2.8  Is  it  Working  and  How  Will  We  Know? 

After  the  implementation  of  NSSAP  03-01  as  interim  guidance,  the  new  policy 
has  already  come  under  close  scrutiny  by  civilian  journals  and  the  GAO.  Some 
opponents  were  critical  of  the  cost  of  space  programs  but  recognized  the  efforts  of  the 
Undersecretary  of  Defense  and  that  he  “has  identified  eight  priorities.”  Among  the  list  is 
“getting  space  acquisition  programs  on  track”  (7:1).  But  the  criticism  still  remains  that 
the  SBR  program  “has  requested  $274  million  and  has  an  estimated  launch  cost  of  $4.4 
billion,  (according  to)  Teets”  (7:2). 

In  one  GAO  study,  a  pre-NSSAP  03-01  review  was  initiated  to  review  the  current 
challenges  facing  space  programs  and  procurement.  They  noted,  that  although  changes 
are  coming,  there  are  still  challenges  facing  this  new  process.  The  GAO  warns  that: 
“Unless  DoD  adopts  knowledge-based  practices,  space  control  acquisitions,  such  as  the 
Space-Based  Surveillance  System,  may  well  face  higher  cost  and  schedule  risks”  (24:2). 
The  warning  incorporates  an  understanding  that  if  a  program  is  not  decentralized  and  a 
streamlined  oversight  process  is  not  accepted  at  all  levels,  the  costs  and  schedule  could  be 
increased  and  prolonged. 

In  another  GAO  study,  after  the  NSSAP  03-01  interim  guidance  was  issued,  the 
GAO  recognized  that  DoD  “was  taking  steps  to  streamline  the  acquisition  process  and 
reduce  the  time  it  takes  to  acquire  space-based  systems  required  by  the  national  security 
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space  community”  (22:14).  The  GAO  stated  that  the  SBR  was  the  first  true  hope  for 
testing  the  new  NSSAP  03-01  process. 

So  how  do  we  know  when  the  oversight  process  is  working  and  how  will  we 
compare  the  cost  of  oversight?  In  the  overview,  the  goal  of  this  chapter  was  presented  as 
showing  how  the  process  changed  over  time  and  how  the  focus  moved  from 
decentralized  to  centralized  oversight  and  the  struggle  between  the  processes.  The 
biggest  problem  was  trying  to  find  a  way  to  compare  the  costs  of  the  oversight  process 
alone.  The  only  document  found  addressing  with  any  calculations  of  actual  oversight 
cost  did  not  provide  any  background  information  or  methodology,  but  did  provide  a  basic 
number  for  comparison.  The  Acquisition  Reform  Process  Action  Team  published  a 
document  in  1994  that  stated  that  a  few  programs  were  tracked  and  the  cost  of  reviews 
was  estimated.  “Our  estimates  suggested  that  the  costs  were  on  the  order  of  $  10-12M, 
including  the  costs  of  a  Cost  and  Operational  Effectiveness  Analysis”  (17:9).  That  initial 
amount  quoted  was  estimated  for  each  milestone  and  “could  add  approximately  $40-50M 
to  a  program  over  its  life”  (17:9). 

Taking  the  estimate  from  the  Process  Action  Team,  and  applying  it  to  the  current 
space  programs  listed  on  the  SAR  equates  to  $35.79  billion  among  seven  programs.  That 
would  estimate  the  cost  of  oversight  to  $30-36M  per  program.  Even  though  this  dollar 
amount  represents  less  than  one  percent  of  the  total  amount  of  programs’  budget,  the 
figures  $210-252M  for  the  total  in  potential  space  MDAP  oversight  a  substantial  amount 
of  resources  that  could  be  spent  on  other  programs. 

This  thesis  will  create  a  foundation  by  using  a  panel  of  experts  to  estimate  the  cost 
of  space  acquisition  oversight  using  the  current  NSSAP  03-01  IPA  process  and  compare 
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these  initial  results  to  those  of  other  acquisition  methods.  The  findings  of  this  thesis 
could  come  up  with  a  potential  measurement  for  any  oversight  savings  by  the  new 
decentralization  process  and  display  any  reluctance  of  experts  to  adopt  the  streamlining 
reforms.  By  identifying  the  main  cost  drivers  and  comparing  with  the  other  acquisition 
methods,  the  potential  of  future  streamlining  efforts  may  be  able  to  focus  on  reducing  or 
eliminating  the  cost  drivers  with  the  potential  of  reducing  the  vertical  layers  in  the 
oversight  process. 
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3.0  Methodology 


3.1  Overview 

From  the  previous  two  chapters,  we  now  have  a  clear  picture  of  the  focus  of  this 
research.  This  research  will  attempt  to  estimate  the  cost  of  oversight  of  Major  Defense 
Acquisition  Programs  (MDAPs)  strictly  under  the  direction  of  the  new  National  Security 
Space  Acquisition  Policy  03-01.  In  addition  to  estimating  the  cost  of  oversight,  the  aim 
of  this  research  is  to  answer  specific  research  questions  outlined  in  Chapter  1 .  Again,  the 
first  and  most  important  question  to  be  answered  is  what  is  the  cost  of  oversight  for  space 
programs?  The  next  question  is  how  does  the  cost  of  oversight  for  space  programs 
compare  to  the  cost  of  oversight  for  MDAPs  operating  under  a  different  framework; 
specifically,  communications  acquisition  programs,  which  are  operating  in  a  “virtual 
box”,  and  “box”  programs  which  are  operating  under  the  DoDD  5000  series?  The  final 
research  question  to  be  answered  is  what  are  the  five  key  drivers  that  affect  the  cost  of 
oversight  of  MDAPs?  As  noted  in  Chapter  2,  the  Delphi  Method  of  surveying  experts 
will  be  employed  to  address  these  questions.  This  chapter  will  outline  the  Delphi 
Method,  how  it  works,  and  how  specifically  the  Delphi  Method  will  be  utilized  for  this 
research. 

3.2  Delphi  Method  Background 

In  this  section,  of  the  chapter,  some  background  information  on  the  Delphi 
Method  is  provided.  After  discussing  the  history  of  the  Delphi  Method,  it  is  important  to 
discuss  what  it  is,  and  finally  describes  how  it  works.  After  discussing  the  history  and 
providing  a  thorough  background,  the  Delphi  Method  will  prove  itself  as  a  perfect  fit  and 
the  chapter  will  close  with  why  the  Delphi  Method  was  the  chosen  methodology  for  this 
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research.  The  methodology  for  the  execution  of  this  current  research  project  will  be 
interspersed  within  the  description  of  each  of  these  subject  areas. 

3.2.1  History  of  the  Delphi  Method 

According  to  Clayton,  the  name  “Delphi”  was  associated  with  Greek  mythology 
and  refers  to  a  Delphi  Oracle  which  was  capable  of  predicting  the  future  (10:376).  The 
Delphi  Method  was  actually  born  in  the  1960s  out  of  the  American  defense  industry  as 
part  of  a  project  called  “Project  Delphi”,  which  was  a  study  conducted  by  the  RAND 
Corporation  in  support  of  an  exploration  by  the  Air  Force  (8:700-701).  The  U.S.  Air 
Force  wanted  to  detennine  what  would  be  key  nuclear  targets  and  what  would  be  the 
likely  number  of  warheads  employed  against  the  United  States  in  the  event  of  nuclear 
attack  by  the  Soviets.  “Project  Delphi”  sought  to  reach  a  consensus  of  expert  opinion  in 
order  to  answer  those  two  critical  questions  from  the  viewpoint  of  a  Soviet  nuclear 
strategist. 

3.2.2  What  is  the  Delphi  Method? 

The  Delphi  Method  is  best  described  as  a  communication  tool  that  facilitates  a 
communication  process  by  allowing  a  group  of  individuals  to  work  as  a  whole  to  deal 
with  a  problem  (8:701).  The  Delphi  Method  attempts  to  reach  a  consensus  of  opinion 
among  the  members  of  the  group,  which  will  here  on  be  referred  to  as  an  expert  panel, 
through  a  series  of  questionnaires.  A  key  element  of  the  questionnaires  is  that  they  are 
completed  anonymously  to  allow  for  freedom  of  expression  and  then  collected, 
summarized  and  returned  to  panel  members  to  give  them  the  opportunity  to  refine 
original  responses  with  the  added  benefit  of  knowing  the  rest  of  the  panel  members’ 
responses.  This  process  is  continued,  “until  consensus  is  obtained  or  the  law  of 
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diminishing  returns  sets  in”  (25: 1010).  Another  key  element  of  the  process  is  that  the 
panel  is  made  up  of  pre-selected  experts  who  never  physically  have  to  be  in  the  same 
location.  The  process,  which  came  into  practice  in  the  1960s,  could  only  be  conducted  by 
traditional  mail,  but  of  course  can  now  be  conducted  via  the  web  or  e-mail,  or  a 
combination  of  both. 

3.2.3  How  the  Delphi  Method  Works 

The  previous  section  of  this  chapter  offered  a  preliminary  look  at  how  the  Delphi 
Method  works,  but  this  section  will  go  into  much  greater  detail  on  the  workings  of  the 
Delphi  Method.  First,  it  is  important  to  answer  some  questions.  The  first  is  why  use  a 
panel  of  experts  that  never  meet  instead  of  just  a  single  expert.  The  reason  is  that  an 
individual  is  operating  along  which  means  they  could  forget  something  or  fail  to  consider 
an  issue.  Clayton  highlighted  this  issue  when  he  discussed  the  fact  that  individuals  don’t 
get  the  benefit  of  hearing  the  ideas  of  others  so  that  they  can  perhaps  refine  their  ideas 
(10:375).  Clayton  goes  on  to  state  that  by  combining  the  judgment  of  a  large  number  of 
people,  there’s  a  better  chance  of  arriving  at  the  truth. 

With  explaining  the  need  for  separated  groups,  a  new  question  arises.  If  a  group 
is  better  than  an  individual,  wouldn’t  it  be  better  to  put  them  in  a  room  together  to  allow 
them  to  brainstorm  and  hammer  out  a  consensus?  Though  this  research  operates  under 
Clayton’s  premise  that  the  shared  ideas  of  a  group  of  experts  is  better  than  a  single 
expert,  putting  a  panel  in  a  room  together  could  lead  to  group  think  (10:375).  This 
phenomenon  is  the  result  of  a  few  dominant  personalities  controlling  the  discussion  and 
potentially  strong  arming  a  consensus  despite  the  initial  objections  of  possibly  better 
informed,  yet  more  timid  panel  members. 
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Now  that  the  two  preliminary  questions  regarding  the  overall  set  up  of  the  Delphi 
Method  have  been  answered,  the  next  step  is  to  describe  the  workings  of  the  Delphi 
Method.  To  aid  in  this  presentation,  the  key  elements  of  the  workings  of  the  Delphi 
Method  are  explained  best  in  Figure  3.1. 


Figure  3.1:  Oversight  Approval  Levels  (16:2) 
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First,  a  problem  is  defined.  For  this  research,  the  research  questions  are  the  main 
problems  defined  which  is  to  determine  the  cost  of  oversight  for  space  programs  as  well 
as  compare  the  costs  of  those  space  programs  to  the  virtual  and  “box”  programs.  The 
other  research  focus,  using  the  Delphi  Method,  is  to  determine  key  oversight  cost  drivers. 
The  next  step  is  to  develop  a  questionnaire  that  is  specific  enough  to  divulge  the  data 
necessary  to  answer  those  questions.  The  following  step  is  to  select  a  panel  of  experts  to 
answer  the  questionnaire.  The  questionnaires  are  then  sent  to  the  panel  and  when 
completed  they  are  collected,  analyzed,  and  summarized.  If  consensus  is  not  reached,  the 
summarized  responses  will  then  be  sent  back  out  to  the  panel  to  allow  them  to  rethink  the 
questions  now  that  they  have  the  added  benefit  of  the  input  from  the  other  group 
members.  This  process  of  sending  out  the  questionnaires  and  then  getting  them  back  and 
analyzing  them  continues  in  a  looping  pattern  and  each  loop  is  referred  to  as  a  “round.” 
Each  time  a  new  questionnaire  is  distributed  marks  the  beginning  of  a  new  round.  The 
number  of  rounds  is  determined  by  the  achievement  of  consensus  of  the  expert’s 
opinions.  Early  criticisms  of  the  Delphi  Method  centered  on  the  fact  that,  due  to  lack  of 
technology,  originally  questionnaires  were  sent  by  traditional  mail  channels.  Depending 
on  the  number  of  rounds  needed  to  achieve  consensus,  the  process  took  from  several 
months  up  to  a  year  or  two  to  complete.  Today’s  technology  enables  the  process  to  flow 
more  quickly,  and  for  the  purpose  of  this  particular  research  effort,  all  communication 
during  the  process  will  be  conducted  via  e-mail.  Chou  takes  this  e-mail  centered  Delphi 
methodology  a  step  further  by  conducting  a  web  based  Delphi  Process  whereby  panel 
members  and  the  survey  director  interact  in  a  shared  web  program  (9:233-236).  In 
summary,  the  Delphi  Method,  as  employed  in  this  research  effort,  will  act  as  a 
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communication  facilitator  that  attempts  to  achieve  a  consensus  of  opinions  from  an 
anonymous,  geographically  separated  panel  of  experts  through  a  series  of  questionnaires 
all  conducted  via  e-mail. 

3.2.3. 1.  The  Rounds  of  the  Delphi  Method 
As  previously  mentioned,  each  time  a  questionnaire  is  distributed  to  panel 
members  and  returned  to  the  person  directing  the  research  effort  constitutes  a  round  of 
the  Delphi  Method.  The  question  that  arises  deals  with  how  many  rounds  of  the  Delphi 
are  necessary  to  ensure  the  data  is  stable.  Clayton  states  that  only  four  phases  are  needed 
and  that  the  final  round  is  sent  out  to  “provide  reasons  as  to  why  they  agree  or  disagree 
with  the  final  results”  (10: 129).  Chan  et  al.  agreed  in  their  study  by  establishing  four 
rounds  (8:701).  However,  Ludwig  states  that  “Delphi  rounds  continue  until  a 
predetennined  level  of  consensus  is  reached  or  no  new  information  is  gained”  (29:3). 
While  a  study  in  Scotland,  by  Dr.  Kerr,  limited  the  number  of  rounds  to  three  (26:3).  In 
recent  nursing  research,  Hasson  et  al.  limited  the  number  of  rounds  depending  on  “time 
available. . .”  (25: 101 1)  The  research  did  not  find  a  specific  number  of  rounds  needed. 
Most  researchers  using  the  Delphi  Method  set  the  criteria  of  consensus  and  time  available 
while  some  limited  on  a  firm  number.  Based  on  the  evidence,  the  Delphi  method  as 
employed  in  this  research  effort  to  answer  the  research  questions,  will  contain  a  minimum 
of  two  rounds  and  a  maximum  of  four. 

3.2.3.2  Delphi  Method  Questionnaires 

Mitchell  goes  into  great  detail  outlining  the  construction  and  administration  of  the 
Delphi  questionnaires.  He  clearly  outlined  the  length  the  questionnaire  should  be  by 
stating  how  long  it  should  take  each  panel  member  to  complete  the  questionnaire.  On 
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this  topic  he  states  that  the  questionnaire  should  take  no  longer  than  30  minutes  to 
complete  (30:345).  The  basis  for  this  assertion  is  his  own  experience  as  he  goes  on  to 
state  that  there  have  been  no  empirical  studies  conducted  on  the  appropriate  length  of 
time  to  complete  a  Delphi  questionnaire.  Mitchell  also  discusses  the  construction  of  the 
questionnaire  for  each  round  of  the  Delphi  Method.  He  states  that  questions  should  be 
clearly  stated  and  should  not  be  identical  from  round  to  round  because  the  repetition 
could  cause  participant  boredom,  which  could  hamper  results  (30:342).  Clayton  also 
discussed  the  format  of  the  questionnaires  on  a  round  by  round  basis.  He  states  that 
round  one  questionnaires  should  be  clearly  worded  but  allow  for  the  most  freedom  in 
responses.  Round  one  responses,  once  collected,  should  be  turned  into  generic 
statements  summarized  with  measures  of  central  tendency  and  then  resent  to  panel 
members  to  begin  round  two.  In  round  two,  the  process  of  seeking  consensus  begins.  To 
aid  in  the  quest  for  consensus,  panel  members  that  wish  to  change  previous  responses 
must  provide  reasons  for  doing  so.  In  round  three  and  subsequent  rounds,  questionnaires 
should  summarized  responses  with  a  summary  of  reasons  for  changing  responses  and  this 
process  continues  until  consensus  is  met  (10:378).  The  questionnaires  in  support  of  this 
research  effort  will  be  constructed  according  to  the  procedures  outlined  by  Clayton  and 
Mitchell.  The  number  of  questions  will  be  limited  to  10  or  less.  The  maximum  amount 
of  time  needed  to  complete  each  questionnaire  is  estimated  at  20  minutes.  Each  returning 
questionnaire’s  questions  are  altered  in  each  round  based  on  the  previous  round’s  input. 
This  will  ensure  each  panel  member  has  the  opportunity  to  re-evaluate  each  question. 
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3.2.3.3  Delphi  Method  Consensus 

The  rounds  of  questionnaires  must  eventually  come  to  a  close.  In  order  to  set  the 
parameters  prior  to  beginning,  once  consensus  is  reached,  the  rounds  will  discontinue. 
Webster’s  New  International  Unabridged  Dictionary  defines  consensus  as,  “unanimity  or 
general  agreement  in  matters  of  opinion”  (52:567).  If  that  definition  is  applied  to  the 
Delphi  Method  as  employed  in  this  research  effort,  once  the  panel  reaches  a  majority 
opinion,  the  process  is  complete;  but  just  a  majority  may  not  be  enough.  Simply 
operating  under  the  theme  of  “majority  rule”  could  overlook  important,  though  less 
frequently  occurring  opinions.  Therefore,  in  terms  of  the  application  of  the  Delphi 
Method  for  this  research  effort,  consensus  must  be  defined.  The  problem,  as  Williams 
and  Webb  state,  “Consensus  is  poorly  explained  in  studies  which  use  the  Delphi 
technique. . .”  (53:182).  Hasson  et  al.  also  state  that  “A  universally  agreed  proportion 
does  not  exist  for  the  Delphi. . .”  (25: 101 1).  Hasson  et  al.  does  list  various  studies  who 
established  percentages  for  defining  consensus,  but  all  vary  dramatically  and  result  in 
mostly  a  straight  majority  rules.  This  study  completed  by  Schiebe  et  al.  recommends 
stability  of  responses  throughout  the  rounds  as  a  better  indicator  of  consensus  by 
evaluating  the  changes  in  the  questions  to  a  quartile  in  a  distribution  (42:IV:C).  Without 
much  empirical  evidence  to  support  a  concrete  definition  of  consensus,  this  research 
effort  will  take  an  approach  similar  to  the  one  recommended  by  Schiebe  et  al.  Each 
question  will  be  evaluated  on  the  response  and  as  answers  become  stable,  the  question 
will  be  considered  “closed”  until  all  questions  are  closed  or  four  rounds  have  been 
completed. 
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3.2.3.4  Delphi  Method  Expert  Panel 

Another  obstacle  when  perfonning  the  Delphi  Method  is  deciding  how  big  the 
expert  panel  should  be.  Spinelli  conducted  research  utilizing  the  Delphi  Method  and  the 
panel  consisted  of  “24  key  influential  persons  knowledgeable  as  to  the  factors  influencing 
the  general  environment. . .”  (44:74).  Ludwig  conducted  research  but  had  a  different 
approach  to  establishing  a  panel.  Ludwig  stated  that  “The  number  of  respondents  was 
generally  detennined  by  the  number  required  to  constitute  a  representative  pooling  of 
judgements  and  the  infonnation  summarizing  capability  of  the  research  team”  (29:2). 

This  establishes  the  precedent  that  as  long  as  all  members  of  the  focus  research  are 
represented,  the  number  of  members  on  the  panel  is  up  to  the  researcher.  Ludwig  then 
states  “The  majority  of  Delphi  studies  have  used  between  15-20  respondents  and  run  over 
periods  of  several  weeks”  (29:2).  Chan  et  al  stated  in  their  selection  process  “The  ten 
members  of  the  panel  represent  a  wide  distribution  of  professional  people. . .”  (8:701). 
Another  study  by  Des  Marchais  reduced  the  panel  size  to  six  (12:504),  but  overall, 
William  and  Webb  summarize  the  panel  selection  methodology  by  stating  “First,  there  is 
no  agreement  regarding  the  size  of  the  panel,  nor  any  recommendations  concerning 
sampling  techniques”  (53:182). 

The  panel  assembled  to  answer  the  research  questions  posed  in  this  thesis  will  be 
of  the  heterogeneous  type  and  will  embody  the  principal  of  breadth  of  members’ 
experience  while  maintaining  the  similar  target  career  field.  The  panel  will  contain  a 
minimum  of  five  and  a  maximum  of  ten  members. 

Once  the  size  of  the  panel  has  been  decided,  establishing  criteria  is  needed  to 
judge  who  the  experts  are.  Based  on  the  findings  that  were  a  result  of  the  research 
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conducted  to  complete  this  chapter,  it  appears  there  is  no  clear  cut  definition  of  what 
constitutes  an  expert.  While  discussing  the  topic  of  expert  panel  member  selection, 
Mitchell  states,  “No  reported  Delphi  study  has  addressed  this  selection  issue”  (30:340). 
Dawson  and  Brucker,  in  their  research,  summarized  the  criteria  for  determining  experts 
used  in  several  Delphi  studies  in  their  field.  The  common  theme  was:  general  experience 
of  seven  years;  specific  experience  of  five  years;  at  least  one  published  article;  at  least 
one  national  conference  presentation;  and  experience  should  be  recent  to  within  the  last 
three  years  (15: 132-134).  For  the  purpose  of  this  research,  those  general  standards  are 
relaxed  by  requiring:  general  experience  of  five  years;  specific  experience  of  two  years; 
recent  experience  within  the  last  five  years;  and  no  qualification  of  presentations  or 
publications. 

Once  the  expert  panel  is  formed,  but  prior  to  the  process  starting,  a  plan  must  be 
instituted  for  panel  attrition.  In  a  study  by  Chan  et  al.  conducted  in  the  field  of  medicine, 
they  achieved  a  response  rate  of  80%  and  went  on  to  state  that  derived  from  various 
studies  that  the  average  response  rate  for  the  medical  field  ranged  from  58%  to  80% 
(8:708).  Mitchell  states  that,  “High  rates  of  attrition  may  mean  that  final  results  are  based 
upon  an  unrepresentative  sub  sample  of  the  original  sample”  (30:341).  To  combat  panel 
attrition  and  the  resulting  degraded  response  rates,  this  research  effort  will  choose  experts 
from  different  but  related  fields  and  have  at  least  one  backup  expert  for  every  expert  so  in 
the  end,  even  with  an  attrition  as  high  as  50%,  all  groups  will  be  represented  and  the  bias 
that  Mitchell  describes  will  be  avoided. 
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3.3  Uses  of  the  Delphi  Method 

The  Delphi  Method  has  had  many  uses  in  research.  According  to  the  book  The 
Delphi  Method:  Techniques  and  Applications,  the  Delphi  Method  was  principally  used  as 
a  forecasting  tool  back  as  early  as  the  1960s  and  went  on  to  say  today  the  Delphi  Method 
is  used  for:  normative  forecast;  to  ascertain  values  and  preferences;  quality  of  life 
estimates;  simulated  and  real  decision  making;  and  inventive  planning.  The  book  also 
went  on  to  state  that  the  Delphi  Method  is  used  extensively  where  “judgmental  input 
data”  is  needed  when  other  data  is  unavailable  or  too  costly  (28:615).  Hasson  et  al.  stated 
that  the  Delphi  Method  is  used  frequently  in  health  and  social  sciences  (25:1008). 
Mitchell’s  article  cites  a  table  listing  the  use  of  the  Delphi  method  by  percentage  by  field 
of  study  from  a  total  of  800  studies.  Delphi  was  most  heavily  used  in  physical  sciences 
and  engineering  (26%  of  all  studies  conducted)  and  the  second  most  frequent  usage  was 
in  business  and  economics  (23%)  (30:334). 

3.4  Criticisms  of  the  Delphi  Method 

If  employed  properly,  the  Delphi  Method  is  an  excellent  tool  for  gathering  data  to 
answer  questions  when  that  data  first  appears  to  be  unavailable.  Since  this  research  effort 
originally  sought  to  analyze  historical  data  and  because  that  data  was  unavailable,  the 
Delphi  Method  appears  to  be  a  suitable  data  collection  method.  There  are  criticisms  to 
bear  in  mind  before  using  the  Delphi  Method.  The  first  criticism  deals  with  who  actually 
decides  what  qualifies  as  an  “expert”.  Clayton  acknowledges  that  expertise  is  not  exactly 
measurable  however,  he  states  that  the  criteria  is  really  relative  based  on  the  peers  of  the 
experts.  For  this  research  effort,  criteria  for  panelists  will  be  based  criteria  found  in  the 
section  on  the  expert  panel  found  in  this  chapter.  Using  Clayton’s  premise  that  experts 
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are  deemed  as  such  by  their  peers,  the  research  will  include  a  preliminary  survey  of 
potential  experts.  Those  providing  experts  will  be  provided  with  our  panel  criteria  and 
asked  whether  they  agree  with  each  of  the  criteria  or  not  and  why. 

Williams  and  Webb  introduce  a  second  criticism  of  the  Delphi  Method  which  is 
that  the  researcher’s  analysis  and  summary  of  each  rounds’  responses  could  introduce 
bias  into  the  process  (53:182).  That  point  is  well  taken  and  to  combat  that  threat, 
responses  will  be  analyzed  using  basic  statistical  methods  (mean,  median,  standard 
deviation)  to  the  fullest  extent  possible.  Additionally,  because  this  research  will  conduct 
the  Delphi  Method  as  part  of  a  group  project,  there  will  be  more  than  one  set  of  eyes 
analyzing  the  responses  which  should  also  help  to  keep  the  process  honest. 

A  final  criticism  of  the  Delphi  Method  regards  the  question  of  reliability; 
specifically,  what  evidence  is  out  there  that  proves  the  Delphi  Method  is  reliable.  In 
other  words,  have  studies  been  conducted  that  prove  findings  were  consistent  in  different 
Delphi  experiments  using  similarly  composed  panels  answering  the  same  questions. 
Williams  and  Webb  found  that,  “there  is  no  evidence  that  the  Delphi  Method  is  reliable” 
(53:182).  Hasson  et  al.  support  these  findings  stating  that  their  research  discovered, 
“There  is  no  evidence  of  the  reliability  of  the  Delphi  Method”  (25: 1012).  Mitchell  stated 
that  other  studies  have  found  a  high  degree  of  replicability  which  would  contradict  the 
criticism  that  the  Delphi  Method  is  unreliable  or  at  the  very  least  unproven  (30:35 1). 

3.5  Strengths  of  the  Delphi  Method 

The  strengths  of  the  Delphi  Method  outweigh  the  drawbacks  previously 
mentioned.  First,  the  Delphi  Method  enables  a  group  of  experts  in  geographically 
separated  locations  to  work  together  without  the  cost  or  other  logistical  problems 
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associated  with  bringing  experts  together  at  a  central  location  (15: 129).  Anyone  who  has 
tried  to  put  together  a  major  conference  would  greatly  appreciate  this  strength. 

The  second  strength  focuses  on  the  fact  that  the  Delphi  Method  results  in  a 
consensus  of  opinion  without  the  bias  or  group  think  that  might  result  from  a  roundtable 
process  (53:181).  This  “anonymous  factor”  ensures  all  panel  members  are  equally 
involved  and  all  panel  members  feel  free  to  answer  honestly.  By  this,  the  researcher  has 
the  opportunity  to  receive  uncensored  answers. 

Williams  and  Webb’s  research  also  highlights  the  Delphi  concept  of  conducting  a 
series  of  rounds  to  achieve  consensus  (53: 181).  The  series  of  rounds  allows  panel 
members  to  review  the  responses  of  their  fellow  panel  members  and  gives  them  the 
chance  to  reconsider  or  even  alter  their  original  responses  with  the  benefit  of  the  added 
input  of  their  fellow  panel  members.  Conducting  only  one  round  would  destroy  the 
intellectual  synergy  created  by  the  sharing  of  ideas  throughout  the  rounds. 

Finally,  a  criticism  of  the  traditional  Delphi  process  that  evolved  into  a  strength 
for  today  was  that  the  traditional  Delphi  process  took  a  long  time  to  complete.  This  long 
period  was  due  to  the  fact  that  it  was  used  in  the  1960s  and  1970s  at  a  time  when  there 
was  no  means  other  than  through  postal  channels  to  conduct  Delphi  rounds.  Chien 
Chou’s  article  highlights  the  final  strength  of  the  Delphi  that  evolved — speed.  Chou 
stated  that  traditional  Delphi  processes  averaged  six  to  twelve  months  from  start  to  finish, 
but  with  e-mail  and  web-based  Delphi  a  three  round  study  can  be  conducted  in  four 
weeks  (9:236). 
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3.6  The  Reason  the  Delphi  Method  was  Chosen 

The  originally  theorized  methodology  for  this  research  effort  was  to  examine  the 
paper  trail  left  by  an  actual  MDAP  going  through  a  milestone  decision  point  i.e.  Meeting 
minutes,  meeting  notes,  sign  in  rosters  to  arrive  at  an  estimated  cost  of  oversight.  Using 
these  documents,  the  ranks  and  number  of  people  at  the  meetings  could  be  ascertained  as 
well  as  the  number  and  duration  of  the  meetings.  This  data  could  then  be  used  to 
estimate  a  cost  of  meetings  based  on  length  of  meeting  and  the  hourly  wages  of  each 
attendee.  The  estimate  for  meeting  costs  at  every  level  of  vertical  oversight  could  then 
be  tallied  to  arrive  at  a  total  estimate  of  the  cost  of  oversight  for  an  MDAP  at  a  certain 
key  decision  point.  The  problem  encountered  with  this  methodology  is  lack  of  data. 
There  was  an  initial  mistake  made  of  assuming  the  meeting  minutes,  notes  and  logs 
would  be  readily  available  when  in  fact  in  some  cases  they  were  nonexistent.  A 
methodology  was  needed  that  would  enable  me  to  answer  the  research  questions  without 
the  availability  of  historical  data.  An  article  by  W.L.  Mitchell  which  appeared  in 
Technology  Analysis  &  Strategic  Management,  outlines  why  one  would  use  the  Delphi 
Method  with  the  number  one  reason  listed  being  the  unavailability  of  historical  data 
(30:338). 

3.7  Comparative  Analysis  for  Data  Collected 

Once  the  rounds  of  the  Delphi  are  completed,  all  data  from  this  study  must  be 
statistically  compared  with  the  data  collected  by  Neal  (32)  for  Command,  Control, 
Communications,  and  Intelligence  (C3I)  and  Rousseau  (40)  for  Department  of  Defense 
Directive  5000  (DoDD  5000).  After  acquiring  their  data,  all  data  for  questions  two 
through  ten  will  be  placed  into  a  statistics  analytical  software  package  with  a  graphical 
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user  interface,  such  as  JMP  5.0.1  statistical  software.  The  data  will  be  entered,  for  each 


question  two  through  ten,  in  the  fonnat  shown  in  Table  3.1. 


Table  3.1:  Data  Input  for  Statistical  Analysis 


Oversight 

2-Low 

2-Avg 

2-High 

NSSAP  03-01 

1 

4 

6 

NSSAP  03-01 

2 

6 

9 

NSSAP  03-01 

1 

4 

7 

NSSAP  03-01 

2 

4 

8 

DoDD  5000 

6 

8 

16 

DoDD  5000 

8 

12 

18 

DoDD  5000 

6 

10 

18 

DoDD  5000 

6 

10 

25 

DoDD  5000 

6 

12 

18 

C3I 

12 

20 

30 

C3I 

7 

9 

12 

C3I 

12 

20 

30 

C3I 

12 

20 

30 

The  format  in  Table  3.1  will  allow  JMP  5.0.1  to  analyze  the  statistical  differences  and 
will  provide  a  value  which  will  test  whether  or  not  there  is  a  statistical  difference  among 
the  different  oversight  processes. 


To  conduct  the  analysis  of  variances,  each  oversight  process  will  compare  with 


one  other  oversight  at  a  time.  For  example,  NSSAP  03-01  will  compare  with  DoDD 


5000.  The  null  hypothesis  for  the  test  is  that  there  is  no  difference  in  the  data  collected. 


The  alternate  is  that  there  is  a  statistical  difference  between  them.  The  test  significance 


for  these  statistical  tests  will  be  set  at  .05. 


The  first  question  will  simply  compare  the  different  cost  drivers  identified  for 


each  specific  oversight  regulating  document  and  provide  insight  to  potential  differences. 


3.8  Summary 
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This  research  effort  is  aimed  at  answering  the  following  research  questions: 


1 .  According  to  experts  in  the  field,  what  is  the  estimated  cost  of  oversight  for  a 
space  acquisition  program  under  the  NSSAP  03-01  IPA  oversight  process? 

2.  How  does  the  cost  of  oversight  for  space  programs  compare  to  the  cost  of 
oversight  for  programs  under  the  DoDD  5000  series  and  the  virtual  acquisition 
processes? 

3.  What  are  the  five  main  cost  drivers  for  the  oversight  of  space  acquisition 
programs? 


This  chapter  outlined  how  this  research  effort  will  answer  those  questions.  In 
summary,  the  research  will  consist  of  assembling  a  panel  of  five  to  ten  experts  in  the  field 
of  defense  acquisition;  prepare  questionnaires  aimed  at  collecting  the  cost  of  oversight  at 
one  key  decision  point  and  aimed  at  identifying  oversight  cost  drivers;  then  employ  the 
Delphi  Survey  technique  of  sending  out  the  questionnaires,  collecting,  analyzing, 
summarizing,  and  resending  questionnaires  to  the  panel;  and  continue  with  the  Delphi 
rounds  until  a  consensus  of  expert  opinion  is  reached.  In  Chapter  4,  the  results  of  each 
round’s  questionnaires  will  be  recorded  and  analyzed. 
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4.0  Data  Results 


4.1  Overview 

The  goal  of  chapter  4  is  to  provide  the  collection  of  the  expert  panel  members  for 
the  Delphi  Method  and  the  generalized  demographics  of  the  panel  members  while  still 
maintaining  the  members’  anonymity.  The  next  section  will  provide  the  results  of  the 
survey  for  each  question.  The  results  will  be  presented  for  one  question  as  it  passed 
through  the  four  separate  rounds  of  the  Delphi  Method.  The  information  provided  will 
include  the  initial  answers  for  each  question  and  how  the  answer  changed  through  the 
rounds  of  the  Delphi  Method.  The  final  section  will  provide  a  summary  of  results  and 
reiterate  the  goal  of  the  Delphi  Method  to  ensure  all  objectives  were  met.  The  first 
research  question  for  the  total  cost  of  oversight  for  space  under  the  NSSAP  03-01  will  be 
answered  in  the  summary  of  results  section.  The  discussion  will  also  include  the  next 
step  on  the  analysis  of  oversight  and  what  will  be  used  to  statistically  compare  these 
results  with  the  data  gathered  by  Neal  (32)  and  Rousseau  (40). 

To  establish  the  cost  of  oversight,  an  algorithm  was  created  with  multiplies  and 
adds  the  respondents  estimates  together  to  create  low,  average,  and  high  estimates  for  the 
cost  of  oversight.  This  is  completed  by  taking  questions  two,  three,  and  four  and 
multiplying  them  together  to  create  a  TDY  cost  estimate.  Then  taking  questions  five,  six, 
and  seven  and  multiplying  them  together  to  create  a  personnel  cost  estimate.  Then, 
questions  eight,  nine,  and  ten  are  multiplied  together  to  create  a  meeting  estimate. 

Finally,  the  TDY,  personnel,  and  meeting  estimate  are  added  together  to  provide  an 
estimate  for  the  cost  of  one  program  to  get  through  one  decision  point.  The  total  program 
cost  for  the  low  estimate  is  then  represented  by  the  following  algorithm: 
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3*((Q21ow*Q31ow*Q41ow)+(Q51ow*Q61ow*Q71ow)+(Q81ow*Q91ow*Q101ow)) 


This  will  repeat  for  the  average  and  high  estimates  as  well. 

4.2  Panel  Selection 

The  goal  of  panel  selection  was  to  gather  experts  in  the  space  acquisition  field, 
but  from  different  viewpoints  in  the  oversight  process.  The  following  individuals,  listed 
on  the  table  below,  were  selected  and  numbered  to  keep  them  anonymous  from  each 
other. 


Table  4.1:  Panel  Selection  Demographics 


Number 

Militarv/Civlian 

Position 

Years  Acq 

Exp 

Years 

Space  Exp 

1 

Civilian 

Senior  GS  Employee 

24 

24 

2 

Military 

Senior  Military  Officer 

19 

13 

3 

Military 

University  Professor  in  Acquisition 

19 

7 

4 

Civilian 

Retired  Military,  Civ  Contractor 

8 

8 

5 

Military 

University  Professor  in  Acquisition 

18 

4 

Since  the  members  are  now  numbered,  the  rest  of  the  results  and  analysis  will 
refer  to  only  the  number  assigned  for  the  Delphi  Method.  As  seen  in  Table  4.1,  there  is  a 
broad  range  of  years  of  experience  as  well  as  the  difference  in  viewpoints  of  the 
acquisition  process.  According  to  the  information  provided  in  the  methodology  section, 
this  will  provide  the  most  probability  of  approaching  the  true  answer  of  the  unknown 
forecast  we  are  trying  to  make  and  compare.  These  members  contain  a  wealth  of 
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experience  and  had  the  desire  to  provide  feedback.  Prior  Internal  Review  Board 
pennission  was  requested  and  obtained  for  this  research  and  the  letter  of  approval  can  be 
seen  in  the  attachment  section. 

When  the  first  round  was  initiated,  panel  member  5  had  to  drop  out  due  to  an 
unforeseen  incident.  Even  without  panel  member  5,  the  heterogeneous  group  was  still 
maintained  and,  according  to  research  conducted  in  chapter  3,  four  members  were  still 
enough  to  conduct  a  Delphi  Method  survey. 

4.3  Question  One 

From  the  Program  Executive  Officer  (PEO)  recommendation,  to  the  Defense 
Systems  Acquisition  Board  (DSAB)  approval  (including  the  entire  Integrated 
Program  Assessment  (IP A)  process),  what  are  the  five  major  cost  drivers  in  the 
oversight  process? 

The  goal  of  question  one  was  to  obtain  the  five  key  cost  drivers  that  the 
respondents  felt  drove  the  cost  of  oversight  higher  than  it  should  truly  be.  These  items 
aren’t  always  budgeted  for  and  may  not  have  true  dollar  figures  attached,  but  could 
potentially  lead  to  increased  costs  in  oversight. 

4.3.1  Results  by  Round 

Round  one  provided  13  items  that  were  thought  to  drive  costs  in  the  oversight 
process.  They  are  listed  below,  in  no  particular  order; 

1 .  Time  away  from  primary  responsibilities  while  supporting  “this”  effort  (IP A)  at 
the  expense  of  the  rest  of  the  program. 

2.  Adding  requirements  to  original  program(from  HQ  &  OSD,  or  other  sources) 

3.  TDY  from  the  program  office  to  IPA  or  IPA  folks  to  program  office 

4.  Hiring  individuals  for  the  IPA  may  come  at  program  office’s  expense 

5.  Time  of  the  program  office  and  PEM  staff  putting  together  packages  as  well  as 
running 

them  through  the  system 

6.  IPA  Personnel  Costs  (Program  evaluation) 

7.  ACE  Personnel  Costs  (Process  oversight) 

8.  SAF/USI  Personnel  Costs  (Process  oversight) 
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9.  IPA  Travel/Per  Diem  costs  (Team  and  support  personnel) 

10.  Facilities  and  Equipment 

1 1 .  Salaries  of  IPA  core  members  and  “gray  beard”  members  who  are  not 
government  employees 

12.  Reproduction  costs  of  the  briefings,  VTCs,  copies,  long  distance  calls,  other 
admin  requirements 

13.  Contractor  cost  of  time  spent  answering  IPA  questions 


With  the  13  responses  from  round  one  collected,  the  drivers  were  compiled  and 
sent  out  in  round  two.  For  round  two,  the  members  were  only  allowed  to  pick  five  from 
the  list  provided.  This  is  the  list  that  was  created  by  the  member  and  their  peers.  The 
round  two  results  are  listed  below,  again,  in  no  particular  order; 

1.  Time  away  from  primary  responsibilities  while  supporting  “this”  effort  (IPA)  at 
the  expense  of  the  rest  of  the  program. 

2.  Adding  requirements  to  original  program(from  HQ  &  OSD,  or  other  sources) 

3.  TDY  from  the  program  office  to  IPA  or  IPA  folks  to  program  office 

4.  Hiring  individuals  for  the  IPA  may  come  at  program  office’s  expense 

5.  Time  of  the  program  office  and  PEM  staff  putting  together  packages  as  well  as 
running  them  through  the  system 

6.  IPA  Personnel  Costs  (Program  evaluation) 

7.  SAF/USI  Personnel  Costs  (Process  oversight) 

8.  IPA  TravePPer  Diem  costs  (Team  and  support  personnel) 

9.  Salaries  of  IPA  core  members  and  “gray  beard”  members  who  are  not 
government  employees 

10.  Contractor  cost  of  time  spent  answering  IPA  questions 

In  round  two,  only  three  drivers  fell  off  from  the  previous  round.  In  round  two  it 
was  noticed  that  most  items  focus  around  paying  non-DoD  members  and  travel  costs. 
The  drivers  eliminated  from  the  previous  round  were  numbers  7,  10,  and  12  from  round 
one. 


Once  again,  these  were  compiled  and  sent  out  for  round  three,  with  the  same 
instructions  of  picking  only  five.  The  answers  for  round  three  focused  on  a  few  specific 
areas,  but  still  had  a  ways  to  go  from  identifying  five  main  drivers.  The  results  from 
round  three  are  listed  below; 
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1.  Time  away  from  primary  responsibilities  while  supporting  “this”  effort  (IPA)  at 
the  expense  of  the  rest  of  the  program. 

2.  Adding  requirements  to  original  program(from  HQ  &  OSD,  or  other  sources) 

3.  TDY  from  the  program  office  to  IPA  or  IPA  folks  to  program  office 

4.  Hiring  individuals  for  the  IPA  may  come  at  program  office’s  expense 

5.  Time  of  the  program  office  and  PEM  staff  putting  together  packages  as  well  as 
running  them  through  the  system 

6.  IPA  Personnel  Costs  (Program  evaluation) 

7.  SAF/USI  Personnel  Costs  (Process  oversight) 

8.  IPA  Travcl/Pcr  Diem  costs  (Team  and  support  personnel) 

9.  Salaries  of  IPA  core  members  and  “gray  beard”  members  who  are  not 
government  employees 

10.  Contractor  cost  of  time  spent  answering  IPA  questions 


As  one  can  begin  to  see,  a  problem  was  beginning  to  develop  and  a  new  way  to 
come  up  with  five  main  drivers  was  needed.  Noticing  that  there  was  a  few  that  were  on 
each  listing,  the  results  were  ranked  by  the  number  of  votes  received.  Three  of  the  items 
had  the  most  selections  and  were  set  aside  as  the  first  three  cost  drivers.  The  remaining 
items  that  received  two  votes  or  more  were  left  for  round  four,  where  the  members  were 
asked  to  only  pick  two  of  the  drivers  listed.  The  new  list  sent  for  round  four  is  shown 
below; 

4.  Hiring  individuals  for  the  IPA  may  come  at  program  office’s  expense 

7.  SAF/USI  Personnel  Costs  (Process  oversight) 

8.  IPA  Travel/Per  Diem  costs  (Team  and  support  personnel) 

9.  Salaries  of  IPA  core  members  and  “gray  beard”  members  who  are  not  government 
employees 


With  instructions  to  only  select  two  from  above,  the  results  were  almost 
unanimous.  All  members  selected  driver  number  9  and  three  out  of  four  selected  number 
8.  With  all  rounds  completed,  the  main  cost  drivers  identified  by  the  Delphi  panel  are 
listed  below,  in  no  particular  order; 

1.  Time  away  from  primary  responsibilities  while  supporting  “this”  effort  (IPA)  at 
the  expense  of  the  rest  of  the  program. 
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2.  TDY  from  the  program  office  to  IPA  or  IPA  folks  to  program  office 

3.  IPA  Personnel  Costs  (Program  Evaluation) 

4.  Salaries  of  IPA  core  members  and  “gray  beard”  members  who  are  not  government 
employees 

5.  IPA  Travel/P erDiem  costs  (Team  and  support  personnel) 


4.4  Question  Two 

From  the  PEO  recommendation,  to  the  DSAB  approval  of  the  KDP,  use  your 
professional  judgment  and  estimate  how  many  TDYs  are  taken  by  one  person  to 
get  one  program  through  one  Key  Decision  Point  (KDP) /Integrated  Program 
Assessment  (IPA)  process? 

The  goal  of  question  two  was  to  find  out  how  many  TDYs  are  taken  by  one 
individual  in  one  program  in  support  of  a  KDP.  The  members  are  asked  to  provide  a 
low,  high  and  average,  or  most  likely  occurrence  for  this  portion.  This  will  allow  us  to 
establish  a  triangular  distribution  that  will  be  used  later  for  the  data  analysis  portion,  as 
well  as  allow  us  to  estimate  the  low,  average,  and  high  costs  of  oversight  for  our 
comparison  of  the  three  different  MDAP  processes. 

Question  two  sets  up  our  initial  number  in  our  algorithm  to  calculate  the  first 
portion  of  our  cost  of  oversight  model.  Question  two,  three,  and  four  will  establish  a 
travel  estimate  for  the  cost  of  oversight. 

4.4.1  Question  Two  -  Low  Estimate 

The  low  estimate  was  established  early  in  the  survey  process.  In  round  one,  the 
results  were  split  directly  down  the  middle.  Two  respondents  answered  1  and  two 
answered  2.  This  did  not  change  at  all  during  the  rest  of  the  survey  process  for  all 
remaining  rounds.  Consensus  was  reached  immediately,  due  to  the  reluctance  of  any 
member  to  change  this  portion  of  the  answer. 

4.4.2  Question  Two  -  Average  Estimate 
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The  average,  or  most  likely,  estimate  was  similar  to  the  low  estimate.  Once  the 
initial  numbers  were  established,  there  was  no  desire  to  change  the  answers  by  any 
respondent.  The  question  was  left  in  for  three  rounds  to  make  sure  that  no  last  minute 
changes  were  made,  but  for  the  fourth  round,  it  was  removed  due  to  the  goals  stated  by 
the  initial  methodology,  which  was  to  minimize  the  standard  deviation,  or  when  the 
results  are  not  being  changed.  The  results  are  listed  in  Table  4.2,  shown  below.  In  the 
column  labeled  “Frequency”  the  answers  are  given  by  numerical  order  for  all  further 
result  tables.  So  the  first  number  is  respondent  one,  the  second  is  respondent  two,  and  it 
continues.  This  will  allow  the  reader  to  view  who  made  what  changes  in  each  round. 


Table  4.2:  Question  Two  -  Average  Estimate  Results  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

4  to  6 

4, 6, 4, 4 

4.5 

4 

1 

2 

4  to  6 

4, 6, 4, 4 

4.5 

4 

1 

3 

4  to  6 

4, 6, 4, 4 

4.5 

4 

1 

4 

Consensus 

Reached 

4.4.3  Question  Two  -  High  Estimate 

The  high  estimate  was  slightly  more  volatile  than  the  low  and  average  estimates 
for  question  two.  The  high  estimate  moved  slightly  on  the  second  round  and  then 
remained  stationary.  After  the  third  round,  consensus  was  declared  and  the  question  was 
closed.  Only  one  member  adjusted  their  response  for  question  two.  The  results  for 
question  two-high  estimate  are  found  below  in  Table  4.3. 


Table  4.3:  Question  Two  -  High  Estimate  Results  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

6  to  10 

6,10,7,8 

7.75 

7.5 

1.70782 

2 

6  to  9 

6, 9, 7, 8 

7.5 

7.5 

1.29099 

3 

6  to  9 

6, 9, 7, 8 

7.5 

7.5 

1.29099 

4 

Consensus 

Reached 
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4.5  Question  Three 


Estimate  how  many  people  normally  go  TDY  throughout  the  KDP/IPA  process. 


Question  three  established  another  portion  of  the  travel  estimate  in  our  cost  of 
oversight  algorithm.  The  goal  of  question  three  is  to  find  the  number  of  personnel  that 
actually  go  TDY  in  an  IPA  process  for  a  KDP.  The  respondents  were  given  the  same 
instructions  as  was  given  with  question  two  and  the  answers  will  be  presented  in  the 
similar  manner. 

4.5.1  Question  Three  -  Low  Estimate 

The  low  estimate’s  standard  deviation  slowly  decreased  through  out  the  Delphi 
process,  shown  in  Table  4.4  below.  Panel  member  1  answered  4  on  the  first  round  and 
changed  their  answer  to  13  in  round  two,  then  didn’t  change  it  again  for  the  remaining 
rounds.  Member  2  answered  20  and  stayed  with  20  through  all  four  rounds,  which  was 
similar  to  member  4,  who  answered  25.  Member  3  moved  their  response  the  most.  Each 
round,  member  3  changed  theirs  closer  and  closer  to  the  average,  which  is  expected  in 
this  type  of  survey.  This  question  was  not  closed  until  all  rounds  were  completed. 


Table  4.4:  Question  Three  -  Low  Estimate  Results  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

3  to  25 

4,20,3,25 

13 

12 

11.16542 

2 

10  to  25 

13,20,10,25 

17 

16.5 

6.78233 

3 

12  to  25 

13,20,12,25 

17.5 

16.5 

6.13731 

4 

13  to  25 

13,20,15,25 

18.25 

17.5 

5.37742 

4.5.2  Question  Three  -  Average  Estimate 

The  average  estimate  for  question  three  went  all  four  rounds  of  the  survey.  The 


results  are  listed  in  Table  4.5  on  the  next  page.  The  standard  deviation  was  decreased  by 


over  50  percent  and  the  range  was  decreased  as  well.  Members  1  and  4  selected  an 
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estimate  and  elected  not  to  change  their  answer  through  all  four  rounds.  Member  2 
changed  their  estimate  on  the  last  round  and  member  3  changed  their  estimate  at  each 
survey  round. 


Table  4.5:  Question  Three  -  Average  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

6  to  30 

20,30,6,30 

21.5 

25 

11.35781 

2 

15  to  30 

20,30,15,30 

23.75 

25 

7.5 

3 

20  to  30 

20,30,25,30 

26.25 

27.5 

4.78713 

4 

20  to  30 

20,25,25,30 

25 

25 

4.08248 

4.5.3  Question  Three  -  High  Estimate 

The  high  estimate  results,  listed  in  table  4.6,  show  almost  the  same  pattern  as  the 
average  results.  Members  1  and  4  selected  an  estimate  and  stuck  with  it  and  member  2 
stayed  with  their  estimate  until  the  last  round,  then  lowered  their  estimate  closer  to  the 
mean  and  median.  Member  3  changed  their  estimate  immediately  and  then  stuck  with  it 
through  the  remaining  rounds.  Overall,  the  standard  deviation  did  decrease,  but  not  as 
much  as  the  average  estimate  for  question  three. 


Table  4.6:  Question  Three  -  High  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

14  to  45 

30,40,14,45 

32.25 

35 

13.67174 

2 

30  to  45 

30,40,30,45 

36.25 

35 

7.5 

3 

30  to  45 

30,40,30,45 

36.25 

35 

7.5 

4 

30  to  45 

30,35,30,45 

35 

32.5 

7.07106 

4.6  Question  Four 

What  is  your  estimate  of  the  cost  for  each  person  on  each  TDY? 

Question  four  provides  the  final  number  for  the  travel  portion  of  the  cost  of 
oversight  formula.  By  multiplying  the  number  from  questions  two,  three,  and  four,  an 
estimate  for  the  cost  of  travel  in  the  oversight  process  can  be  obtained.  Question  four 
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will  provide  an  actual  dollar  figure  estimate  for  the  cost  of  one  TDY  for  one  person  on  a 


team.  Results  are  presented  in  the  same  format  as  previous  questions. 

4.6.1  Question  Four  -  Low  Estimate 

Question  four,  low  estimate  showed  a  pattern  of  a  constantly  decreasing  standard 


deviation.  The  decrease  in  the  standard  deviation  was  significant  in  coming  to  an  almost 


full  consensus  on  the  low  estimate.  All  panel  members  made  some  changes  in  their 


estimate  at  one  point  or  another,  which  can  be  seen  below  in  Table  4.7. 


Table  4.7:  Question  Four  -  Low  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

500  to  1000 

$1000/500/700/850 

762.5 

775 

213.6009 

2 

700  to  1000 

$1000/750/700/700 

787.5 

725 

143.61407 

3 

700  to  750 

$750/  750/  700/  700 

725 

725 

28.86751 

4 

700  to  750 

$750/  750/  725/  700 

731.25 

737.5 

23.93567 

4.6.2  Question  Four  -  Average  Estimate 

Results,  listed  in  Table  4.8,  show  the  initial  range  and  standard  deviation  were 
quite  large  for  the  first  round.  The  initial  assumption  was  that  this  would  decrease 
significantly  over  the  next  rounds.  Members  1  and  4  set  their  initial  estimates  and  did  not 
change  through  the  rounds,  but  members  2  and  3  did  change  their  estimates.  Member  3 
changed  their  estimate  once  and  left  it  alone  and  member  2  changed  the  estimate  three 
out  of  four  rounds.  Although  the  standard  deviation  did  not  decrease  to  an  actual  number 
that  most  would  consider  small,  it  did  decrease  by  over  50  percent,  which  could  be 
considered  significant,  given  the  data  set  to  analyze. 


Table  4.8:  Question  Four  -  Average  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

800  to  2000 

$1800/800/  2000/  1200 

1450 

1500 

550.75705 

2 

1200  to  1800 

$1800/  1200/  1500/  1200 

1425 

1350 

287.22813 

3 

1200  to  1800 

$1800/  1300/  1500/  1200 

1450 

1400 

264.57513 

4 

1200  to  1800 

$1800/  1300/  1500/  1200 

1450 

1400 

264.57513 
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4.6.3  Question  Four  -  High  Estimate 

The  high  estimate  for  question  four  had  a  large  range,  seen  in  Table  4.9,  and 
similar  to  the  average  estimate,  a  large  standard  deviation.  Member  1  stood  fast  in  their 
estimate  and  never  changed  the  high  estimate.  Member  2  changed  their  estimate  in 
rounds  two  and  four  while  members  3  and  4  only  changed  their  estimates  in  round  two 
and  then  left  them  alone.  The  standard  deviation  decreased  to  32  percent  of  the  original 
standard  deviation. 


Table  4.9:  Question  Four  -  High  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

1000  to  5000 

$3600/  1000/  5000/  2000 

2900 

2800 

1762.5739 

2 

2200  to  3600 

$3600/2800/  3000/  2200 

2900 

2900 

577.35027 

3 

2200  to  3600 

$3600/2800/  3000/  2200 

2900 

2900 

577.35027 

4 

1200  to  1800 

$3600/2900/  3000/  2200 

2925 

2950 

573.73048 

4.7  Travel  Computation 

When  taking  each  value  provide  by  each  panel  member  and  multiplying,  as  stated 
in  the  first  section  of  our  algorithm,  the  calculation  becomes  the  first  section  in  the  cost  of 
oversight  estimate  for  the  NSSAP  03-01  oversight  process.  These  amounts  are  seen  in 
Table  4.10.  The  range,  from  low  to  high,  are  $9,750  to  $913,000. 


Table  4.10:  Total  Travel  Estimate  for  Cost  of  Oversight  for  One  KDP 


Member 

Travel-Low 

Travel-Average 

Travel-High 

1 

$9,750 

$144,000 

$648,000 

2 

$30,000 

$195,000 

$913,500 

3 

$10,875 

$150,000 

$630,000 

4 

$35,000 

$144,000 

$792,000 

4.8  Question  Five 


Estimate  how  many  hours  are  spent  on  support  for  the  DSAB  approval  process 
per  person,  not  including  TDY  travel  time,  but  actual  job  performance  while 
TDY  or  at  home  base,  (slide  prep,  meeting  prep,  etc) 
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In  question  five,  a  new  segment  of  the  cost  of  oversight  algorithm  is  started. 
Question  five  is  the  beginning  of  the  personnel  portion  of  the  estimate.  With  question 
five,  the  goal  is  to  find  the  number  of  hours  personnel  put  in  directly  towards  the  DSAB 
process. 

4.8.1  Question  Five  -  Low  Estimate 


The  results,  shown  in  Table  4. 1 1,  started  off  with  another  large  range.  In  the 
round  two,  all  members,  but  member  4,  changed  their  estimates.  Members  1  and  2  cut 
their  estimates  by  half  and  member  3  raised  theirs  450  percent.  Member  4  made  an 
estimate  and  stayed  with  that  estimate.  Since  the  answers  did  not  change  in  round  three, 
it  was  considered  closed  and  consensus  reached.  It  was  assumed  that  no  other  changes 
were  made  which  would  cause  any  other  member  to  change  their  answer. 


Table  4.11:  Question  Five  -  Low  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

10  to  120 

120,160,10,44 

83.5 

82 

68.67071 

2 

44  to  80 

60,80,45,44 

57.25 

52.5 

16.83993 

3 

44  to  80 

60,80,45,44 

57.25 

52.5 

16.83993 

4 

Consensus 

Reached 

4.8.2  Question  Five  -  Average  Estimate 

In  Table  4. 12,  the  average  estimates  have  almost  the  same  pattern  as  the  low 
estimate  for  question  five.  This  time,  the  only  members  to  change  any  responses  were 
members  2  and  3  and  they  only  changed  the  estimates  initially.  Consensus  was  reached 
due  to  the  similar  circumstances  stated  in  the  low  estimate  portion  of  question  five. 

Table  4.12:  Question  Five  -  Average  Estimate  By  Round 
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Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

20  to  200 

180,200,20,120 

130 

150 

80.82903 

2 

90  to  180 

180,120,90,120 

127.5 

120 

37.74917 

3 

90  to  180 

180,120,90,120 

127.5 

120 

37.749174 

4 

Consensus 

Reached 

4.8.3  Question  Five  -  High  Estimate 

Table  4.13  below  shows  the  results  for  the  high  estimates.  Once  again  these 
showed  the  similar  pattern  as  the  earlier  portions  of  question  five.  The  range  decreased 
by  over  half  and  the  standard  deviation  decreased  by  over  half.  The  members  did  not 
show  any  movement  in  changing  estimates  from  round  two  to  round  three,  so  it  was 
determined  that  consensus  was  reached  for  the  high  estimate  of  question  five. 


Table  4.13:  Question  Five  -  High  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

45  to  300 

240,300,45,200 

196.25 

220 

108.88641 

2 

135  to  240 

240,160,135,200 

183.75 

180 

46.07512 

3 

135  to  240 

240,160,135,200 

183.75 

180 

46.07512 

4 

Consensus 

Reached 

4.9  Question  Six 

Estimate  how  many  people  are  normally  involved  with  the  preparation  process. 

Question  six  places  an  actual  number  of  personnel  into  the  second  portion  of  the 
algorithm  for  cost  of  oversight.  The  number  of  personnel  involved  in  the  preparation 
process  included  those  creating  slides,  preparing  briefings,  and  supporting  the  IPA.  The 
results  are  given  in  similar  format  as  previous  data  collected. 

4.9.1  Question  Six  -  Low  Estimate 

The  low  estimate,  seen  in  Table  4. 14,  was  a  volatile  subject.  Almost  every 
respondent  changed  their  estimate,  except  member  4,  who  chose  nine  and  stuck  with  that 
estimate  throughout.  The  rest  of  the  panel  agreed  with  member  4  and  came  closer  and 
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closer  to  that  estimate.  The  standard  deviation  was  reduced  to  less  than  one,  which  is 


considered  a  close  consensus. 


Table  4.14:  Question  Six  -  Low  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

4  to  40 

4,40,3,9 

14 

6.5 

17.53062 

2 

5  to  10 

9,10,5,9 

8.25 

9 

2.21735 

3 

7  to  10 

9,10,7,9 

8.75 

9 

1.2583 

4 

9  to  10 

9,10,10,9 

9.5 

9.5 

0.57735 

4.9.2  Question  Six  -  Average  Estimate 

The  average  estimate,  located  in  Table  4.15,  showed  less  movement  than  other 
estimates.  The  mean  and  the  median  stayed  constant  while  the  standard  deviation  moved 
in  round  two.  After  round  two,  no  other  movement  was  made  and  the  average  estimate 
was  not  changed.  Due  to  the  change  in  the  low  estimate,  however,  question  six  remained 
open  until  the  end  of  round  four. 


Table  4.15:  Question  Six  -  Average  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

5  to  50 

20,50,5,35 

27.5 

27.5 

19.36491 

2 

20  to  35 

20,30,25,35 

27.5 

27.5 

6.45497 

3 

20  to  35 

20,30,25,35 

27.5 

27.5 

6.45497 

4 

20  to  35 

20,30,25,35 

27.5 

27.5 

6.45497 

4.9.3  Question  Six  -  High  Estimate 

The  high  estimate,  seen  below  in  Table  4.16,  is  rather  stable  from  the  start  of 
round  one  through  the  end  of  round  four.  There  were  still  changes  made  to  the  estimates 
by  all  panel  members.  Members  1  and  4  changed  their  estimates  initially  and  then  never 
changed  the  estimate  again.  Member  2  changed  their  estimate  in  round  two  and  then 
reduced  the  estimate  in  round  4.  Member  3  increased  their  estimate  during  each  round. 
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In  the  final  round  the  mean  and  median  were  stable  and  the  standard  deviation  was 


reduced  significantly  from  the  initial  round. 


Table  4.16:  Question  Six  -  High  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

12  to  76 

30,70,12,76 

47 

50 

31.00537 

2 

40  to  55 

40,55,40,50 

46.25 

45 

7.5 

3 

40  to  55 

40,55,42,50 

46.75 

46 

6.99404 

4 

40  to  50 

40,50,45,50 

46.25 

47.5 

4.78713 

4.10  Question  Seven 

Estimate  the  cost  per  hour  for  each  person  involved  in  the  process. 

Question  Seven  provides  the  last  portion  of  the  personnel  cost  estimate  for  the 
total  cost  of  oversight.  With  the  estimates  provided  in  question  seven,  multiplied  by  the 
estimates  given  in  questions  five  and  six,  the  estimated  forecast  for  the  cost  of  personnel 
in  the  oversight  process  is  projected. 


4.10.1  Question  Seven  -  Low  Estimate 


In  Table  4.17,  the  low  estimates  are  provided.  The  estimates  changed  in  the  first 


three  rounds  and  remained  unchanged  for  the  final  round.  The  standard  deviation  was 


reduced  and  the  median  remained  constant  after  round  two.  Member  1  and  4  remained 


with  their  initial  estimate  during  all  four  rounds.  Member  2  changed  their  estimate  in 


round  two  and  member  3  changed  their  estimate  in  rounds  one,  two  and  three. 


Table  4.17:  Question  Seven  -  Low  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

32  to  100 

55.5 

45 

30.56686 

2 

32  to  75 

$32, $50, $75, $50 

51.75 

50 

17.67059 

3 

32  to  50 

$32, $50, $50, $50 

45.5 

50 

9 

4 

32  to  50 

$32, $50, $50, $50 

45.5 

50 

9 

4.10.2  Question  Seven  -  Average  Estimate 
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Question  seven’s  estimates,  located  in  Table  4.18,  came  to  a  consensus  in  round 


three  with  a  minimized  standard  deviation  of  2.5.  It  was  continued  due  to  the  potential 


decrease  in  round  four.  Member  4  was  the  only  member  who  maintained  a  constant 


estimate  while  other  members  appeared  to  converge  to  member  4’s  estimate.  The  range 


went  from  eighty  to  five  in  three  rounds. 


Table  4. 18:  Question  Seven  -  Average  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

50  to  130 

$60, $50, $130, $75 

78.75 

67.5 

35.67795 

2 

60  to  100 

$60, $70, $100, $75 

76.25 

72.5 

17.01714 

3 

70  to  75 

$75, $70, $75, $75 

73.75 

75 

2.5 

4 

70  to  75 

$75, $70, $75, $75 

73.75 

75 

2.5 

4.10.3  Question  Seven  -  High  Estimate 

The  high  estimate  for  question  seven,  seen  in  Table  4.19,  did  not  change  as  much 
s  the  average  estimates.  Although  the  standard  deviation  did  not  change  a  considerable 
amount,  it  was  still  decreased  by  50  percent  and  the  range  was  decreased  by  from  190  to 
100.  Member  1  stayed  with  their  original  estimate  during  all  four  rounds  and  was  the 
highest.  Member  2  increased  their  estimate  during  all  four  rounds  and  members  3  and  4 
made  minor  changes  initially. 


Table  4.19:  Question  Seven  -  High  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

60  to  250 

■HHB 

165 

175 

81.03497 

2 

100  to  250 

$250, $100, $200, $175 

181.25 

187.5 

62.5 

3 

130  to  250 

$250, $130, $190, $175 

186.25 

182.5 

49.56056 

4 

150  to  250 

$250, $150, $190, $175 

191.25 

182.5 

42.5 

4.11  Personnel  Cost  Computation 


Calculating  questions  five,  six,  and  seven  together  provide  the  cost  estimate  for 


the  total  personnel  costs  for  one  KDP  in  a  program.  Seen  below  in  Table  4.20,  the 
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figures  are  ranging  from  $17,280  to  $2,400,000  for  personnel  costs  in  the  NSSAP  03-01 
oversight  process. 


Table  4.20:  Total  Personnel  Estimate  for  Cost  of  Oversight  for  One  KDP 


Member 

Personnel-Low 

Personnel-Avg 

Personnel-High 

1 

$17,280 

$115,200 

$2,400,000  I 

2 

$40,000 

$180,000 

$1,200,000 

3 

$22,500 

$112,500 

$1,154,250 

4 

$19,800 

$210,000 

$1,750,000 

4.12  Question  Eight 


Estimate  how  many  meetings  are  normally  held  from  the  PEO  preparation, 
through  DSAB  approval,  (this  includes  meetings  TDY  or  TDY prep  meetings). 


Question  eight  provides  insight  on  how  meetings  are  included  into  the  oversight 
process.  By  multiplying  questions  eight,  nine,  and  ten,  we  will  get  an  idea  of  truly  what 
part  meetings  play  in  the  cost  of  oversight.  Question  eight  deals  specifically  with  the 
number  of  meetings  that  are  held  during  one  KDP  in  a  program.  The  results  are  listed  in 
the  following  three  paragraphs. 


4.12.1  Question  Eight  -  Low  Estimate 

The  low  estimate,  in  Table  4.21,  was  immediately  moved  in  round  two  and  then 
remained  constant  until  survey  completion.  The  standard  deviation  showed  significant 
reduction,  even  for  only  being  changed  once.  The  range  was  decreased  from  twenty  to 
three  and  members  1  and  4  were  the  pace  setters,  once  again,  by  creating  an  estimate  and 
staying  with  that  estimate  throughout  all  four  rounds. 

Table  4.2 1 :  Question  Eight  -  Low  Estimate  By  Round 
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Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

5  to  25 

7,25,5,8 

11.25 

7.5 

9.25112 

2 

7  to  10 

7,10,7,8 

8 

7.5 

1.41421 

3 

7  to  10 

7,10,7,8 

8 

7.5 

1.41421 

4 

7  to  10 

7,10,7,8 

8 

7.5 

1.41421 

4.12.2  Question  Eight  -  Average  Estimate 

The  average  estimate,  seen  below  in  Table  4.22,  fell  mostly  in  between  15  and  16. 
The  standard  deviation  was  decreased  to  25  percent  of  the  original  value.  The  response 
range  was  decreased  from  twenty-seven  to  seven,  with  some  hesitation.  The  round  three 
values  were  not  changed  and  almost  considered  a  consensus,  but  in  round  four,  member  3 
changed  their  estimate  to  approach  the  mean  and  median  of  the  other  respondents. 


Table  4.22:  Question  Eight  -  Average  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

8  to  35 

8,35,10,13 

16.5 

11.5 

12.50333 

2 

12  to  20 

16,20,12,13 

15.25 

14.5 

3.59397 

3 

12  to  20 

16,20,12,13 

15.25 

14.5 

3.59397 

4 

13  to  20 

16,20,14,13 

15.75 

15 

3.09569 

4.12.3  Question  Eight  -  High  Estimate 

The  high  estimate  for  question  eight,  seen  in  Table  4.23,  remained  a  moving 
target  during  all  four  rounds.  The  range  began  at  35  and  decreased  to  13  at  the  end  of 
round  four.  The  standard  deviation  was  decreased  to  approximately  30  percent  of  the 
original  value.  Member  4  stayed  with  their  original  estimate  during  all  four  rounds  and 
members  2  and  3  changed  their  estimates  to  match  closer  with  the  mean  and  median  for 
each  round. 


Table  4.23:  Question  Eight  -  High  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

15  to  50 

16,50,20,15 

25.25 

18 

16.64081 

2 

15  to  35 

25,35,22,15 

24.25 

23.5 

8.3016 

3 

15  to  30 

25,30,23,15 

23 

23.5 

6.27162 

4 

15  to  28 

25,28,25,15 

23.25 

25 

5.6789 
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4.13  Question  Nine 


What  do  you  estimate  as  the  length,  in  hours,  for  each  meeting? 

Question  nine  provides  a  length  for  each  meeting,  which  will  be  multiplied  by  the 
number  of  meetings  provided  in  question  eight  and  the  cost  per  hour  for  each  person, 
provided  in  question  ten.  Question  nine  was  an  estimate  that  came  in  with  relatively  low 
standard  deviations  for  each  estimate,  but  overall,  was  not  significantly  volatile  from  one 
round  to  the  next.  Results  are  provided  in  the  following  three  sub-paragraphs. 

4.13.1  Question  Nine  -  Low  Estimate 

The  low  estimate  in  question  nine,  located  in  Table  4.24,  contained  only  one 
change  by  one  respondent  in  all  four  rounds.  Member  2  changed  their  answer  in  the 
second  round  and  no  other  changes  were  made.  Even  with  the  minor  change  in  the 
estimate  by  member  2,  the  data  set  remained  close  with  a  low  standard  deviation  and  a 
comparatively  close  range. 


Table  4.24:  Question  Nine  -  Low  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

.25  to  4 

.25, 4,1, .3 

1.3875 

0.65 

1.775 

2 

.25  to  2 

.25, 2,1, .3 

0.8875 

0.65 

0.81687 

3 

.25  to  2 

.25, 2,1, .3 

0.8875 

0.65 

0.81687 

4 

.25  to  2 

.25, 2,1, .3 

0.8875 

0.65 

0.81687 

4.13.2  Question  Nine  -  Average  Estimate 

The  average  estimate  for  question  nine,  listed  in  Table  4.25,  had  a  similar  pattern 
to  the  low  estimate.  Members  1  and  4  provided  an  estimate  and  never  changed  their 
initial  estimate.  Member  2  changed  their  initial  estimate  in  the  second  round  and  never 
changed  the  estimate  in  the  following  rounds.  Member  3  changed  the  estimate  in  rounds 
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2  and  4.  In  the  last  round  of  the  survey,  although  the  range  and  standard  deviation  did  not 
noticeably  change,  almost  all  members  came  to  an  agreement  on  an  estimate. 


Table  4.25:  Question  Nine  -  Average  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

2  to  6 

4, 6, 2, 2. 5 

3.625 

3.25 

1.79698 

2 

2.5  to  4 

4, 4, 3, 2. 5 

3.375 

3.5 

0.75 

3 

2.5  to  4 

4, 4, 3, 2. 5 

3.375 

3.5 

0.75 

4 

2.5  to  4 

4, 4, 4, 2. 5 

3.625 

4 

0.75 

4.13.3  Question  Nine  -  High  Estimate 

The  high  estimate  for  question  nine,  located  in  Table  4.26  below,  never  changed 
in  range  and  the  change  in  standard  deviation  was  very  minuscule.  There  was  an  initial 
close  consensus  in  the  first  round  and  slowly  spread  out  during  all  four  rounds.  It  was 
interesting  that,  even  though  three  out  of  four  estimated  eight  on  the  first  round,  by  the 
end  of  round  four,  each  member  had  a  different  estimate.  The  standard  deviation  did 
decrease  due  to  the  closer  interval  to  the  mean  in  the  data  set. 


Table  4.26:  Question  Nine  -  High  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

8  to  12 

12,8,8,8 

9 

8 

2 

2 

8  to  12 

12,8,8,8 

9 

8 

2 

3 

8  to  12 

12,9,8,8 

9.25 

8.5 

1.89296 

4 

8  to  12 

12,9,10,8 

9.75 

9.5 

1.70782 

4.14  Question  Ten 

What  is  the  cost  per  hour  of  each  person  involved  in  the  meetings? 

Question  ten  provided  the  cost  per  person  to  include  in  the  final  portion  of  the 
cost  of  oversight  estimate  for  meetings  conducted  for  a  KDP.  The  results  of  question  ten 
mirrored  the  results  of  question  seven,  due  to  both  dealing  with  the  cost  of  personnel  per 
hour.  The  results  will  still  be  provided  separately  due  to  future  discussion  on  the  cost  of 
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oversight  and  the  analysis  portion  of  the  thesis.  The  estimates  will  be  provided  in  the 
same  format  as  previous  questions. 

4.14.1  Question  Ten  -  Low  Estimate 

Question  ten’s  low  estimate,  listed  in  Table  4.27,  moved  significantly  in  relation 
to  the  range  and  standard  deviation.  Member  3  was  the  only  respondent  to  change  their 
estimate  while  the  other  three  members  left  their  estimate  the  same  from  round  one 


throughout  the  whole  survey. 


Table  4.27:  Question  Ten  -  Low  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

32  to  100 

55.5 

45 

30.56686 

2 

32  to  75 

49.25 

45 

18.67931 

3 

32  to  50 

$32, $40, $50, $50 

43 

45 

8.71779 

4 

32  to  50 

$32, $40, $45, $50 

41.75 

42.5 

7.67571 

4.14.2  Question  Ten  -  Average  Estimate 

The  average  estimate,  seen  in  Table  4.28,  initially  had  a  large  range  of  70  and  a 
relatively  large  standard  deviation  which  was  half  of  the  range.  The  standard  deviation 
was  eventually  reduced  by  round  three  to  7.5  and  most  members  came  to  a  consensus 
with  one  number.  Member  2  changed  their  estimate  in  the  second  round  and  left  the 
estimate  the  same  while  member  3  changed  their  estimate  twice. 


Table  4.28:  Question  Ten  -  Average  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

60  to  130 

$60, $50, $130, $75 

78.75 

67.5 

35.67795 

2 

60  to  100 

$60, $60, $100, $75 

73.75 

67.5 

18.87458 

3 

60  to  75 

$75, $60, $75, $75 

71.25 

75 

7.5 

4 

60  to  75 

$75, $60, $75, $75 

71.25 

75 

7.5 

4.14.3  Question  Ten  -  High  Estimate 
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The  high  estimate,  listed  in  Table  4.29,  was  an  estimate  that  never  really  came  to 
a  consensus.  Member  1  chose  one  number  and  it  was  the  highest  estimate  in  the 
responses.  The  other  members  changed  their  estimates,  at  least  once,  which  approached 
the  mean  and  median  of  the  most  current  round.  The  standard  deviation  did  reduce  by 
almost  half  and  the  data  range  was  also  reduced  by  almost  half. 


Table  4.29:  Question  Ten  -  High  Estimate  By  Round 


Round 

Range 

Member  1,2, 3, 4 

Mean 

Median 

Std  Deviation 

1 

60  to  250 

$250,  $60, $250, $150 

177.5 

200 

91.42392 

2 

100  to  250 

$250, $100,  $200,  $250 

181.25 

187.5 

62.5 

3 

130  to  250 

$250, $130, $200, $175 

188.75 

187.5 

50.06246 

4 

150  to  250 

$250, $150, $150, $175 

181.25 

162.5 

47.32423 

4.15  Meeting  Cost  Computation 

Combining  questions  eight,  nine,  and  ten  provide  the  total  estimated  cost  of 
meetings  in  the  NSSAP  03-01  oversight  process.  This  calculation  is  located  in  Table 
4.30,  seen  below.  The  costs  for  meetings  range  from  only  $56  to  $75,000  for  one  person 
for  each  KDP  in  one  program. 


Table  4.30:  Meeting  Estimate  for  Cost  of  Oversight  for  One  Person 


Member 

Meetings-Low 

Meetings-Avg 

Meetings-High 

1 

$56 

$4,800 

$75,000 

2 

$800 

$4,800 

$37,800 

3 

$315 

$1,200 

$37,500 

4 

$120 

$2,438 

$21,000 

The  numbers  provided  above  only  give  the  cost  for  one  person  going  to  meetings. 
With  these  calculations,  another  computation  is  needed  and  that  requires  the  actual 
number  of  people  attending  the  meetings.  These  were  provided  in  an  additional  question 
at  the  end  of  the  Delphi  to  capture  a  basic  estimate.  The  results  are  located  in  Table  4.3 1 . 
Table  4.31:  Total  Number  of  Members  Present  at  Meetings 
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Member# 

Low 

Avg 

High 

1 

8 

25 

75 

2 

5 

10 

45 

3 

8 

25 

75 

4 

25 

55 

80 

With  this  additional  infonnation,  the  number  of  people  attending  the  meeting  is 


multiplied  with  the  total  cost  of  meetings  to  provide  us  with  a  true  cost  of  meetings  in  the 


oversight  process  under  the  NSSAP  03-01  guidelines.  These  costs  are  seen  in  Table  4.32 


and  range  from  $448  to  $5,625,000. 


Table  4.32:  Total  Cost  of  Meetings  in  Oversight  Process 


MEMBER 

Meeting-LOW 

Meeting-AVG 

Meeting-HIGH 

1 

$448.00 

$120,000.00 

$5,625,000.00 

2 

$4,000.00 

$48,000.00 

$1,701,000.00 

3 

$2,520.00 

$105,000.00 

$2,812,500.00 

4 

$3,000.00 

$134,062.50 

$1,680,000.00 

4.16  Summary  of  Results 

The  goal  of  the  Delphi  Method  was  to  complete  at  least  four  rounds  while  trying 
to  reach  consensus.  The  objectives  were  clear  for  how  consensus  would  be  determined. 
The  rule  was  met  for  all  10  questions  provided  in  the  survey  and  all  objectives  for  the 
data  collection  portion  were  met. 

The  focus  now  turns  to  answering  the  first  research  question  stated  in  Chapter  3. 
This  question  is  to  find  the  cost  of  oversight  for  one  program’s  cost  of  oversight  under  the 
NSSAP  03-01  directives.  By  using  the  algorithm  provided  the  total  amounts  are  seen 
below  in  Table  4.33.  The  forecasted  range  for  the  cost  of  oversight  for  one  program  in 
one  KDP  ranges  from  $27,478  to  $8,673,000.  These  points  represent  the  optimistic  and 
pessimistic  views  of  the  total  cost  of  oversight. 

Table  4.33:  Total  Cost  of  Oversight  for  One  KDP  in  One  Program 
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Key  Decision  Point  (KDP) 

MEMBER 

MDP  Low 

MDP  Avg 

MDP  High 

1 

$27,478.00 

$534,000.00 

$8,673,000.00 

2 

$74,000.00 

$495,000.00 

$3,814,500.00 

3 

$35,895.00 

$423,750.00 

$4,596,750.00 

4 

$57,800.00 

$593,062.50 

$4,222,000.00 

Since  Table  4.33  shows  the  cost  for  only  one  KDP,  by  using  one  of  the 


assumptions  stated  in  Chapter  3,  these  numbers  are  simply  multiplied  by  three  to  provide 


a  forecast  for  the  total  cost  of  oversight  for  a  program  operating  within  the  guidelines  of 


the  NSSAP  03-01.  These  are  shown  below  in  Table  4.34.  The  range  is  $82,434  to 
$26,019,000. 


Table  4.34:  Total  Cost  of  Oversight  for  One  Program 


Total  Program 

MEMBER 

Program  Low 

Program  Avg 

Program  High 

1 

$82,434.00 

$1,602,000.00 

$26,019,000.00 

2 

$222,000.00 

$1,485,000.00 

$11,443,500.00 

3 

$107,685.00 

$1,271,250.00 

$13,790,250.00 

4 

$173,400.00 

$1,779,187.50 

$12,666,000.00 

Now  that  each  member’s  estimates  have  been  provided,  the  information  will  be 
placed  in  statistical  software  as  a  database  in  order  to  answer  the  final  two  research 
questions.  Each  respondent  will  have  their  estimates  entered  for  each  question.  This  will 
be  compared  with  other  respondents  from  the  theses  research  conducted  by  Neal  (32)  and 
Rousseau  (40).  When  comparing  all  of  the  estimates  together,  an  analysis  of  variance  test 
will  be  conducted  by  question,  by  type  of  regulatory  guidance  policy  programs  typically 
fall  under,  (ie,  NSSAP  03-01,  DoDD  5000,  or  C3I) 
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Once  this  analysis  has  been  completed,  the  results  and  analysis  will  be  presented 
in  Chapter  5  to  see  if  there  truly  is  a  difference  in  the  cost  of  oversight  among  programs. 
Along  with  the  comparative  analysis,  a  simulation  will  be  conducted  with  Monte  Carlo 
simulation  techniques.  By  using  the  estimates  provided  by  the  expert  panel  members,  the 
additional  10,000  iterations  will  provide  a  more  accurate  picture  of  the  cost  of  oversight 
of  a  program  under  the  NS  SAP  03-01. 
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5.0  Analysis 


5.1  Overview 

The  goal  of  Chapter  5  is  to  compare  the  results  of  the  final  round  of  Delphi 
surveys  for  each  of  the  acquisition  disciplines  examined.  The  first  section  will  contain  a 
question  by  question  statistical  comparison  of  responses.  The  comparisons  are  conducted 
in  the  manner  discussed  in  the  methodology  described  in  Chapter  3.  Question  one  will 
contain  only  a  qualitative  discussion  on  the  cost  drivers  identified,  but  will  list  all  drivers 
to  complete  the  goal  of  research  question  three.  Questions  two  through  ten  are 
quantitatively  compared  with  a  significance  level  of  .05  for  testing  the  null  hypothesis  of 
finding  any  statistical  differences  in  the  mean,  for  the  forecast  data  collected.  Each 
question  will  include  a  discussion  on  where  the  differences  are  and  discuss  some  of  the 
similarities  among  the  different  disciplines,  answering  research  question  two. 

The  final  section  will  conclude  the  analysis  with  the  recommendation  for  future 
oversight  transformation  efforts  as  well  as  provide  any  insight,  gained  from  this  research, 
as  to  whether  the  goal  of  transformation  realized  by  the  Commission  on  Space  has  been 
achieved.  Finally,  any  future  research  efforts  that  could  continue  to  build  on  this  thesis 
will  be  provided  to  assist  in  defining  the  cost  of  oversight  in  MDAPs  in  the  future 

5.2  Question  One 

From  the  Program  Executive  Officer  (PEO)  recommendation,  to  the  Defense 
Systems  Acquisition  Board  (DSAB)  approval  (including  the  entire  Integrated 
Program  Assesment  (IPA)  process),  what  are  the  five  major  cost  drivers  in  the 
oversight  process? 

The  results  for  all  three  oversight  processes  are  listed  in  Table  5.1  below.  DoDD 
5000  and  C3I  drivers  focused  mainly  on  issues  dealing  with  the  program,  itself,  not  so 
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much  on  the  oversight  process.  This  seems  to  be  due  to  the  oversight  process  for  both  are 
centralized  and  all  aspects  for  the  program  are  centrally  driven  and  approved.  Notice  that 
for  the  NSSAP  03-01  drivers,  they  are  issues  dealing  more  with  lower  level  problems, 
such  as  TDYs,  personnel  out  of  the  program  supporting  other  programs,  or  IP  As.  The 
hypothesis  gained  from  this  perspective  is  that  when  a  program  is  under  a  decentralized 
oversight  process,  the  external  drivers,  such  as  requirements  and  other  issues  are  not  as 
big  of  focus  as  when  a  program  is  in  a  centralized  form  of  oversight.  Although  C3I  is  a 
new  model  for  oversight,  it  is  still  under  the  guidelines  of  the  basic  DoDD  5000  structure 
and  the  drivers  are  noticeably  similar.  This  display  then  answers  research  question  three 
and  identifies  the  main  cost  drivers,  as  seen  by  those  in  the  actual  programs  regulated. 

Table  5.1:  Cost  Drivers  for  Oversight  Processes 


Drivers  Picked-DoDD  5000 

Rank 

Program  is  Multi-Service 

1 

Whether  Completely  new  system  or  just  block  upgrade 

2 

NumberofTechnologies  going  into  the  system 

3 

N urn  ber  of  System s  the  System  m ust  interact  with 

4 

Milestone  B  (reauires  most  documents;  30  to  be  generated  for  review) 

5 

Drivers  Picked-Soace 

Rank 

Time  away  from  primary  responsibilities  while  supporting  IPA  at  expense  of  rest  of  program 

1 

TDY  from  the  program  office  to  IPA  or  IPA  folks  to  program  office 

2 

IPA  Personnel  Costs  (Program  Evaluation) 

3 

Salaries  of  IPA  core  members  and  "gray  beard"  members  who  are  not  government  employees 

4 

IPA  Travel/PerDiem  costs  (Team  and  support  personnel) 

5 

Drivers  Picked--C3l 

Rank  i 

Lack  of  functional  requirements  that  are  clearly  defined  and  understood 

1 

Negotiating  viewpoints  of  the  various  stakeholders. ..acq  strategy  re-do 

2 

The  serial  process  of  document  approval  by  the  several  echelons  of  oversight 

3 

The  lack  of  established  architectures  and  the  resulting  need  for  unique  C4ISP  efforts 

4 

Changing  oversight  requirements;the  way  we  did  things  previously  not  work  now  due  to 
changing  personalities,  policy  etc.— requires  climbing  the  learning  curve  again 

5 
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5.3  Question  Two 

From  the  PEO  recommendation,  to  the  DSAB  approval  of  the  KDP,  use  your 
professional  judgment  and  estimate  how  many  TDYs  are  taken  by  one  person  to 
get  one  program  through  one  Key  Decision  Point  (KDP) /Integrated  Program 
Assessment  ( IPA )  process? 

When  comparing  question  two,  there  were  statistical  differences  for  all  three 
oversight  processes  when  comparing  C3I  and  DoDD  5000  with  Space  oversight.  Seen  in 
Table  5.2,  at  all  levels  of  the  forecasts,  not  one  of  the  estimates  were  statistically  similar 
to  Space  oversight  under  the  NS  SAP  03-01.  When  looking  at  the  data,  Space  was  always 
lower  in  the  estimates  for  the  number  of  TDYs  taken  by  a  person  in  the  program.  With 
this  comparison,  it  is  shown  that,  under  NSSAP  03-01,  it  is  perceived  that  the  IPA 
process  actually  requires  fewer  TDYs  for  each  person  for  each  KDP. 


Table  5.2:  ANOVA  for  Question  Two 


Question  2 

p-Values  < 

(.05  significance  level) 

COMPARISON 

LOW 

AVG 

HIGH  | 

Space  vs  C3I 

0.0004 

0.0038 

0.0075 

Space  vs  5000 

0.0001 

0.0005 

0.0004 

5.4  Question  Three 

Estimate  how  many  people  normally  go  TDY  throughout  the  KDP/IPA  process. 


In  question  three,  there  were  not  any  significant  statistical  differences  when 
looking  for  the  number  of  people  that  actually  go  TDY  in  the  KDP/IPA  process.  Table 
5.3  shows  that,  among  the  different  oversight  processes,  there  aren’t  any  differences  in 
the  perception  that  the  number  of  people  going  TDY  has  decreased.  This  is  especially 
interesting  when  reviewing  the  C3I  process.  The  expectation  would  be  that,  since  it  is  a 
mostly  virtual  process,  the  actual  number  of  people  going  TDY  would  decrease,  however, 
when  looking  at  the  raw  data,  it  is  slightly  lower,  than  DoDD  5000  and  Space,  but  not 
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significant  enough  to  have  a  statistical  difference.  The  shaded  areas  on  each  table 
represents  areas  that  have  no  statistical  difference. 


Table  5.3:  ANOVA  for  Question  Three 


Question  3 

p-Values  (.05  significance  level) 

COMPARISON 

LOW 

AVG 

HIGH 

Space  vs  C3I 

0.0811 

0.1071 

0.1345 

Space  vs  5000 

0.4934 

0.3418 

0.2323 

Fail  to  reject 


5.5  Question  Four 

What  is  your  estimate  of  the  cost  for  each  person  on  each  TDY? 

In  trying  to  capture  the  cost  for  each  person  on  each  TDY,  the  results  were  varied 
for  the  different  oversight  process  comparisons.  Seen  in  Table  5.4,  the  low  estimate 
showed  statistical  significance  in  the  difference  in  forecasts  while  the  average  showed 
very  little  differences,  although  the  average  for  space  and  C3I  were  borderline.  For  the 
low  estimate,  C3Is  costs  were  actually  a  bit  higher  than  the  estimates  for  space  oversight. 
Comparing  the  low  estimates  with  the  DoDD  5000,  the  estimates  are  even  lower.  The 
only  guess  could  be  the  costs  increase  because  the  length  of  the  TDY  is  actually  longer, 
due  to  the  IPA  process.  This  would  cause  the  costs  to  increase,  but  the  actual  number  of 
TDYs,  shown  above  in  Table  5.2,  to  remain  lower. 

The  average  costs  were  statistically  similar.  The  high  estimates  for  cost  of  each  TDY 
was  only  different  between  DoDD  5000  and  Space  oversight.  Again,  based  on  the  initial 
hypothesis  that  the  IPA  TDYs  are  longer,  the  costs  are  higher.  DoDD  5000  estimates 
came  in  actually  lower  than  the  Space  NSSAP  03-01  estimates. 


Table  5.4:  ANOVA  for  Question  Four 
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Question  4 

p-Values  (.05  significance  level) 

COMPARISON 

LOW 

AVG 

HIGH 

Space  vs  C3I 

0.0008 

0.0582 

0.1143 

Space  vs  5000 

0.043 

0.3136 

0.0087 

5.6  Travel  Computation 

Looking  at  all  data  points  combined  for  all  three  theses,  Table  5.5  displays  the 
ranges  forecasted  for  the  total  combinations  of  the  cost  of  travel  in  the  different  oversight 
processes.  Although  in  the  earlier  comparisons  of  travel  estimates,  space  oversight  was 
higher  in  cost  per  TDY,  the  overall  range  of  total  costs  is  lower.  This  again  is 
hypothesized  that  due  to  the  IPA  process,  the  cost  of  each  TDY  is  increased,  but  the  total 
number  of  TDYs  are  decreased,  resulting  in  an  overall  decrease  in  total  TDY  costs.  This 
seems  to  hold  true  when  actually  comparing  the  total  cost  for  travel  among  the  oversight 
processes. 


81 


Table  5.5:  Travel  Cost  Computations 


5000  -  T ravel 

MEMBER 

T  ravel-LOW 

T  ravel-AVG 

Travel-HIGH 

1 

$336,000.00 

$1,152,000.00 

$3,840,000.00 

2 

$576,000.00 

$1,728,000.00 

$6,480,000.00 

3 

$28,800.00 

$192,000.00 

$720,000.00 

4 

$12,000.00 

$100,000.00 

$1,250,000.00 

5 

$18,000.00 

$180,000.00 

$720,000.00 

MEAN 

$194,160.00 

$670,400.00 

$2,602,000.00 

STD  DEV 

$253,711.90 

$732,320.15 

$2,524,048.34 

Space 

MEMBER 

T  ravel-LOW 

Travel-AVG 

Travel-HIGH 

1 

$9,750.00 

$144,000.00 

$648,000.00 

2 

$30,000.00 

$195,000.00 

$913,500.00 

3 

$10,875.00 

$150,000.00 

$630,000.00 

4 

$35,000.00 

$144,000.00 

$792,000.00 

MEAN 

$21,406.25 

$158,250.00 

$745,875.00 

STD  DEV 

$12,979.70 

$24,662.72 

$133,206.84 

C3I  -  T ravel 

MEMBER 

Travel-LOW 

Travel-AVG 

Travel-HIGH 

1 

$36,000.00 

$400,000.00 

$3,000,000.00 

2 

$42,000.00 

$94,500.00 

$432,000.00 

3 

$60,000.00 

$400,000.00 

$2,100,000.00 

4 

$240,000.00 

$1,200,000.00 

$4,800,000.00 

MEAN 

$94,500.00 

$523,625.00 

$2,583,000.00 

STD  DEV 

$97,534.61 

$473,355.99 

$1,821,086.49 

5.7  Question  Five 


Estimate  how  many  hours  are  spent  on  support  for  the  DSAB  approval  process 
per  person,  not  including  TDY  travel  time,  but  actual  job  performance  while 
TDY  or  at  home  base,  (slide  prep,  meeting  prep,  etc) 


Question  five  results,  located  in  Table  5.6,  shows  that  there  were  statistical 
differences  across  every  aspect  for  each  estimate  level.  Comparing  the  C3I  estimates, 
due  to  the  virtual  nature  inherent  in  the  process,  the  hours  for  preparation  are  lower  than 
space  oversight.  This  is  assumed  to  be  due  to  the  availability  of  all  documents  for 
oversight  on  the  internet  or  website.  For  space  oversight,  this  is  not  the  case  and  the 
estimates  for  low,  average,  and  high  are  all  higher  for  space  than  for  C3I. 
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When  comparing  with  DoDD  5000,  the  case  is  an  extreme  opposite.  DoDD  5000 
process  is  significantly  higher  than  space  oversight.  This  is  assumed  to  be  due  to  the 
number  of  meetings  that  are  required  between  each  level  of  approval,  which  is  not  in  the 
IPA  process  for  space  oversight  under  the  direction  for  the  NSSAP  03-01.  This  is  a  very 
significant  question  that  shows  how  much  of  an  influence  the  virtual  process  in  C3I  can 
have  if  possibly  introduced  in  other  programs. 


Table  5.6:  ANOVA  for  Question  Five 


Question  5 

p-Values  (.05  significance  level) 

COMPARISON 

LOW 

AVG 

HIGH 

Space  vs  C3I 

0.0019 

0.0012 

0.0007 

Space  vs  5000 

0.0071 

0.001 

0.0005 

5.8  Question  Six 

Estimate  how  many  people  are  normally  involved  with  the  preparation  process. 

This  comparison  was  split  exactly  down  the  center,  by  oversight  process.  Shown 
in  Table  5.7,  the  difference  between  space  and  C3I  is  once  again  statistically  different  in 
all  three  estimates.  However,  there  are  no  statistical  differences  between  space  and  the 
DoDD  5000  oversight  processes.  When  looking  at  the  raw  data,  C3I  is  definitely  lower 
than  the  other  oversight  processes  when  reviewing  the  number  of  people  involved  in  the 
preparation  process.  Once  again,  the  assumption  is  based  on  the  availability  of  the 
information  to  everyone  involved  in  real-time  and  the  ability  to  view  that  information 
whenever  possible. 

Comparing  space  to  DoDD  5000,  they  are  statistically  similar  due  to  the 
necessary  meeting  preparation  for  IPA  or  DAB  approval.  Although  the  DoDD  5000 
oversight  process  has  an  increased  number  of  meetings,  the  IPA  is  assumed  to  still  have 
those  meetings,  but  only  in  a  set  timeframe  during  the  IPA.  By  showing  this  information, 


83 


it  is  hypothesized  that  the  information  is  still  being  gathered  and  prepared  in  similar 


manners  under  DoDD  5000  and  the  NSSAP  03-01  processes,  while  C3I  enables  the  real¬ 


time  review  for  the  approval  process,  decreasing  the  number  of  people  needed  to  prepare 


for  the  approval  process. 


Table  5.7:  ANOVA  for  Question  Six 


Question  6 

p-Values  (.05  significance  level) 

COMPARISON 

LOW 

AVG 

HIGH 

Space  vs  C3I 

0.0001 

0.002 

0.0001 

Space  vs  5000 

0.2225 

0.2213 

0.1415 

5.9  Question  Seven 

Estimate  the  cost  per  hour  for  each  person  involved  in  the  process. 

Question  seven  was  varied  among  the  three  different  processes,  which  is  seen 
below  in  Table  5.8.  Looking  at  the  low  estimate  first,  the  level  for  Space  compared  with 
C3I  is  statistically  different.  But  when  compared  with  the  DoDD  5000,  there  isn’t  a 
difference.  This  is  again  true  for  the  average  estimate.  But  for  the  high  estimate,  the  two 
swap  and  have  statistical  differences  in  different  order. 

The  data  reviewed  shows  that  the  estimate  for  the  C3I  process  was  higher  in  the 
low  and  average  estimates  when  compared  with  space  and  the  high  estimate  was  very 
similar.  When  reviewing  the  DoDD  5000  data,  the  reverse  is  true.  No  assumption  can  be 
made  as  to  why  this  occurs,  but  the  statistical  differences  from  the  Delphi  processes 
remain. 


Table  5.8:  ANOVA  for  Question  Seven 
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Question  7 

p-Values  (.05  significance  level) 

COMPARISON 

LOW 

AVG 

HIGH 

Space  vs  C3I 

0.003 

0.0001 

0.2178 

Space  vs  5000 

0.6153 

0.0746 

0.0018 

5.10  Personnel  Cost  Computation 

When  reviewing  the  total  estimates  provided  by  all  three  types  of  oversight,  the 
comparison  of  total  costs  for  personnel  show  large  value  differences.  Shown  in  Table 
5.9,  the  C3I  portion  is  significantly  smaller  than  the  NSSAP  or  DoDD  5000  oversight 
processes.  DoDD  5000  oversight  process  has  the  largest  range  for  all  three.  Ranging 
from  $120,000  to  over  $24M,  personnel  costs  create  a  large  portion  for  the  cost  of 
oversight  in  programs  under  the  centralized  control  of  DoDD  5000.  Space  is  estimated 
lower  at  $17,280  to  $2.4M,  but  doesn’t  compare  with  the  C3I  estimates  of  $4,200  to  just 
over  $144,000.  This  shows  that  using  the  C3I  approach  could  provide  some  potential 
cost  savings  in  the  oversight  process  by  placing  items  in  a  virtual  environment  and 
allowing  those  who  have  access  the  ability  to  view  at  their  leisure. 
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Table  5.9:  Personnel  Cost  Computation 


5000  -  Personnel 

MEMBER 

Person-LOW 

Person-AVG 

Person-HIGH 

1 

$2,373,600.00 

$7,654,400.00 

$14,664,000.00 

2 

$2,250,000.00 

$7,312,500.00 

$22,440,000.00 

3 

$182,320.00 

4 

$120,000.00 

$1,200,000.00 

$7,200,000.00 

5 

$215,000.00 

$805,000.00 

$2,820,000.00 

MEAN 

$1,028,184.00 

$10,055,052.00 

STD  DEV 

$1,173,086.83 

$3,574,565.45 

$8,408,165.41 

Space  Personnel 

MEMBER 

Person-LOW 

Person-AVG 

Person-HIGH 

1 

$17,280.00 

$270,000.00 

$2,400,000.00 

2 

3 

$22,500.00 

$168,750.00 

$1,154,250.00 

4 

$19,800.00 

$315,000.00 

$1,750,000.00 

MEAN 

$24,895.00 

$251,437.50 

$1,626,062.50 

STD  DEV 

$10,293.11 

$61,161.80 

$582,659.43 

C3I  -  Personnel 

MEMBER 

Person-LOW 

Person-AVG 

Person-HIGH 

1 

$4,200.00 

$25,500.00 

$135,000.00 

2 

$7,200.00 

$25,200.00 

$129,600.00 

3 

$7,200.00 

$90,000.00 

4 

$7,680.00 

$33,600.00 

$144,000.00 

MEAN 

$6,570.00 

$28,575.00 

$124,650.00 

STD  DEV 

$1,596.12 

$4,005.31 

$23,851.42 

5.11  Question  Eight 

Estimate  how  many  meetings  are  normally  held  from  the  PEO  preparation, 
through  DSAB  approval,  (this  includes  meetings  TDY  or  TDY prep  meetings). 


This  comparison,  located  in  Table  5.10,  found  one  statistical  significance  when 
comparing  the  means  in  the  ANOVA  test.  The  low  estimate  for  the  comparison  of  Space 
to  C3I  had  a  significant  statistical  difference.  Although  the  other  comparisons  did  not 
show  any  differences  at  the  .05  significance  level,  there  were  still  some  notable 
differences  when  reviewing  the  main  data  collected.  It  was  expected  that  the  IPA  process 
would  decrease  the  number  of  meetings  required,  which  was  the  same  as  expected  with 
the  C3I  process.  This  assumption  was  true,  if  one  only  looked  at  the  data,  without  any 
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statistical  testing.  Reviewing  the  data  provided  in  the  DoDD  5000  process  showed  a 
numerically  larger  number  of  meetings  in  the  range  provided,  but,  did  not  develop  a 
difference  when  tested  at  the  .05  significance.  This  will  result  in  a  larger  dollar  figure  for 
the  total  cost  of  oversight  projected  for  the  DoDD  5000  oversight  process,  but  does  not 
show  a  statistical  difference  when  compared  with  the  NSSAP  03-01  process  for  Space. 


T able  5.10:  ANOV A  for  Question  Eight 


Question  8 

p-Values  (.05  significance  level) 

COMPARISON 

LOW 

AVG 

HIGH 

Space  vs  C3I 

0.0372 

0.1248 

0.1306 

Space  vs  5000 

0.0667 

0.1117 

0.0768 

5.12  Question  Nine 


What  do  you  estimate  as  the  length,  in  hours,  for  each  meeting? 

The  length  of  meetings  was  similar  in  almost  all  comparisons.  The  only 


difference,  seen  in  Table  5. 1 1,  is  when  comparing  Space  to  DoDD  5000  oversight.  This 


was  due  to  DoDD  5000  estimates  being  a  little  higher  than  Space.  It  wasn’t  numerically 


different,  but  when  comparing  the  means,  there  was  an  apparent  statistical  difference. 


Otherwise,  comparing  with  the  rest  of  the  estimates,  nothing  significant  was  noted  for 


question  nine. 


T able  5.11:  ANOVA  for  Question  Nine 


Question  9 

p-Values  (.05  significance  level) 

COMPARISON 

LOW 

AVG 

HIGH 

Space  vs  C3I 

0.0601 

1.00 

0.6963 

Space  vs  5000 

0.0346 

0.3451 

0.4016 

5.13  Question  Ten 
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What  is  the  cost  per  hour  of  each  person  involved  in  the  meetings? 


Question  ten  was  varied  among  the  three  different  processes,  which  are  seen  in 
Table  5.12.  Looking  at  the  low  estimate  first,  the  level  for  Space  compared  with  C3I  is 
statistically  different.  But  when  compared  with  the  DoDD  5000,  there  isn’t  a  difference. 
This  is  again  true  for  the  average  estimate.  But  for  the  high  estimate,  the  two  swap  and 
have  statistical  differences  in  different  order. 

The  data  reviewed  shows  that  the  estimate  for  the  C3I  process  was  higher  in  the 
low  and  average  estimates  when  compared  with  space  and  the  high  estimate  was  very 
similar  for  all  three.  When  reviewing  the  DoDD  5000  data,  the  reverse  is  true.  No 
assumption  can  be  made  as  to  why  this  occurs,  but  the  statistical  differences  from  the 
Delphi  processes  remain  and  provide  similar  results  as  those  found  in  question  seven, 
which  dealt  with  the  similar  topic. 


Table  5.12:  ANOVA  for  Question  Ten 


Question  10 

p-Values  (.05  significance  level) 

COMPARISON 

LOW 

AVG 

HIGH 

Space  vs  C3I 

0.0028 

0.0001 

0.1008 

Space  vs  5000 

0.1722 

0.2536 

0.0049 

5.14  Meeting  Cost  Computation 

When  comparing  the  actual  cost  estimate  calculations,  an  interesting  point  came 
to  light.  Shown  in  Table  5.13,  the  total  cost  for  meetings  was  actually  lowest  in  the 
DoDD  5000  oversight  process.  More  information  would  be  required  to  make  any  large 
assumptions  as  to  why  this  had  occurred,  but  the  assumption  that  could  be  made  is  due  to 
the  lower  cost  per  hour  of  personnel  contributing  in  the  meeting  process.  The  other  areas 
for  the  cost  computation  come  in  significantly  higher  in  the  DoDD  portion  when 
comparing  to  the  other  oversight  processes.  Because  of  this  one  factor,  the  C3I  process 
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still  looks  as  though  it  could  statistically  come  in  with  a  lower  cost  for  meetings  than  the 
DoDD  5000  oversight  process.  The  other  interesting  occurrence  is  the  high  costs  that 
were  calculated  in  the  C3I  process.  They  were  associated  with  one  high  point  that  may 
have  been  just  an  anomaly,  but  there  isn’t  sufficient  evidence  to  keep  this  data  point  out. 


Table  5.13:  Meeting  Cost  Computation 


5000  -  Meeting 

MEMBER 

Meeting-LOW 

Meeting-AVG 

Meeting-HIGH 

1 

$51,600.00 

$397,440.00 

$3,384,000.00 

2 

$34,560.00 

$270,000.00 

$2,736,000.00 

3 

$43,756.80 

$260,160.00 

$2,075,220.00 

4 

$7,200.00 

$128,000.00 

$1,800,000.00 

5 

$15,480.00 

$88,320.00 

$406,080.00 

MEAN 

$30,519.36 

$228,784.00 

$2,080,260.00 

STD  DEV 

$18,747.83 

$123,506.69 

$1,119,283.55 

Space  -  Meeting 

MEMBER 

Meeting-LOW 

Meeting-AVG 

Meeting-HIGH 

1 

$448.00 

$120,000.00 

$5,625,000.00 

2 

$4,000.00 

$48,000.00 

$1,701,000.00 

3 

$2,520.00 

$105,000.00 

$2,812,500.00 

4 

$3,000.00 

$134,062.50 

$1,680,000.00 

MEAN 

$2,492.00 

$101,765.63 

$2,954,625.00 

STD  DEV 

$1,495.64 

$37,757.05 

$1,857,179.46 

C3I  -  Meeting 

MEMBER 

Meeting-LOW 

Meeting-AVG 

Meeting-HIGH 

1 

$33,600.00 

$825,000.00 

$14,400,000.00 

2 

$18,000.00 

$112,500.00 

$448,000.00 

3 

$37,500.00 

$300,000.00 

$2,208,000.00 

4 

$39,600.00 

$257,040.00 

$3,686,400.00 

MEAN 

$32,175.00 

$373,635.00 

$5,185,600.00 

STD  DEV 

$9,771.51 

$311,415.24 

$6,283,940.47 

5.15  Summary  of  Results 

Overall,  the  three  research  questions  that  were  stated  as  goals  for  this  thesis  have 
been  answered.  The  total  cost  of  oversight  has  been  calculated  as  an  estimate  for  the  IPA 
process  under  the  NSSAP  03-01  oversight  process.  These  results  are  shown  in  Table 
5.14.  The  cost  drivers  for  all  oversight  processes  have  been  identified.  Finally,  when 
compared  to  the  other  processes,  the  research  question  dealing  with  any  statistical 
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differences  in  the  cost  of  oversight  between  the  different  oversight  processes  has  been 
answered  for  all  but  one  combination;  total  cost  comparison. 


Table  5.14:  Program  Cost  Computation 


5000  -  Program 

MEMBER 

Program  Low 

Program  Avg 

Program  High 

1 

$8,283,600.00 

$27,611,520.00 

$65,664,000.00 

2 

$8,581,680.00 

$27,931,500.00 

$94,968,000.00 

3 

$764,630.40 

$4,039,380.00 

$17,839,440.00 

4 

$417,600.00 

$4,284,000.00 

$30,750,000.00 

5 

$745,440.00 

$3,219,960.00 

$11,838,240.00 

MEAN 

$3,758,590.08 

$13,417,272.00 

$44,211,936.00 

STD  DEV 

$4,270,333.53 

$13,109,978.82 

$35,217,742.48 

Space  -  Program 

MEMBER 

Program  Low 

Program  Avg 

Program  High 

1 

$82,434.00 

$1,602,000.00 

$26,019,000.00 

2 

$222,000.00 

$1,485,000.00 

$11,443,500.00 

3 

$107,685.00 

$1,271,250.00 

$13,790,250.00 

4 

$173,400.00 

$1,779,187.50 

$12,666,000.00 

MEAN 

$146,379.75 

$1,534,359.38 

$15,979,687.50 

STD  DEV 

$63,337.35 

$213,056.30 

$6,761,137.82 

C3I  -  Program 

MEMBER 

Program  Low 

Program  Avg 

Program  High 

1 

$221,400.00 

$3,751,500.00 

$52,605,000.00 

2 

$201,600.00 

$696,600.00 

$3,028,800.00 

3 

$314,100.00 

$2,190,000.00 

$13,194,000.00 

4 

$861,840.00 

$4,471,920.00 

$25,891,200.00 

MEAN 

$399,735.00 

$2,777,505.00 

$23,679,750.00 

STD  DEV 

$311,948.30 

$1,682,759.26 

$21,431,855.48 

The  remaining  task  is  summarizing  the  results  is  to  see  if  there  are  any  statistical 


differences  among  the  oversight  processes  for  the  total  cost  of  one  program  through  the 


whole  process.  The  result  of  this  analysis  is  shown  in  Table  5.15.  The  results  indicate 


that,  given  our  assumptions,  there  is  not  a  statistical  difference  among  the  different 


oversight  processes.  Even  with  the  large  ranges  in  the  estimated  cost  of  oversight  for  the 
different  processes,  they  cannot  be  identified  as  statistically  different. 

Table  5.15:  ANOVA  for  Total  Cost  Comparison 
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Total  Cost 

p-Values  (.05  significance  level) 

COMPARISON 

LOW 

AVG 

HIGH 

Space  vs  C3I 

0.1625 

0.1932 

0.5188 

Space  vs  5000 

0.1393 

0.117 

0.1629 

5000  vs  C3I 

0.1655 

0.1557 

0.343 

The  ranges  for  each  process  can  be  seen  in  Table  5.16,  and  the  overlap  in  the 
range  numbers  explains  the  failure  to  find  significant  differences  in  the  oversight 
processes.  Although  not  statistically  different,  the  results  indicate  that  each  oversight 
process  has  its  own  cost  savings  potential. 

Table  5.16:  Total  Cost  Ranges  by  Oversight  Process  within  Range 


Process 

Low  Range 

Avg  Range 

high  Range 

NSSAP 

$82,434  to  $222,000 

$1,271 ,000  to  $1,779, 188 

$11, 443, 500  to  $26, 019, 000 

C3I 

$201, 000  to  $861 ,840 

$696,600  to  $4,471 ,920 

$3,028,800  to  $52,605,000 

5000 

$41 7, 600  to  $8, 581, 680 

$3,21 9,960  to  $27,931 ,500 

$1 1,838, 240  to  $94, 968, 000 

5.16  Recommendations 


To  enhance  the  oversight  process  and  decrease  the  potential  cost  of  oversight  that 
all  three  oversight  processes  posses,  it  is  my  recommendation  that  the  process  of  IPA  and 
C3I  be  merged  into  an  oversight  process  that  allows  approval  by  an  independent  board, 
but  infonnation  for  those  meetings  and  approval  processes  should  be  available  over  a 
virtual  process.  This  could  potentially  decrease  the  above  costs,  listed  in  Table  5.16  to  a 
smaller  range  located  in  Table  5.17.  By  combining  both  processes,  the  total  for  potential 
savings  ranges  from  $40,000  to  a  little  over  $74M  per  program.  These  funds  could  be 
realized  either  directly  or  indirectly,  either  in  saved  man-hours  or  actual  bottom-line 
budget  savings.  Either  way,  this  move  has  a  potential  for  significant  savings  to  the  point 
that  some  sort  of  live  program  test  or  feasibility  study  should  be  performed. 


Table  5.17:  Proposed  Combined  Oversight  Process  IPA/C3I 
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Process 

Low  Range 

Avg  Range 

High  Range 

Proposal 

$43,194  to  $140,040 

$652,500  to  $1,087,988 

$7,200,000  to  $20,046,000 

Although  the  potential  does  exist  for  these  cost  saving  areas,  it  should  also  be 
noted  that  these  changes  would  require  wide  structural  transformations.  The  review 
process  would  almost  become  an  additional  duty,  instead  of  a  full-time  position.  This 
would  raise  several  questions  on  the  best  structure  for  this  merged  oversight  that  has  been 
proposed. 

5.17  Follow-on  Possibilities 

There  is  a  potential  for  further  research  in  this  area.  An  analysis  gathering  more 
panel  members  and  the  inclusion  of  the  Ballistic  Missile  Defense  Agency  is  a  possibility 
that  could  add  even  more  possibilities  of  cost  savings  for  the  Department  of  Defense  and 
the  oversight  of  future  acquisition  processes.  It  is  also  important  to  note  that  these 
experts,  from  all  three  panels,  were  mostly  from  Air  Force  sources  or  backgrounds.  A 
excellent  follow-on  would  gather  data  from  other  services  and  compare  to  this  study. 

Another  option  is  to  gather  additional  data  and  utilizing  simulation  to  increase  the 
number  of  data  points  collected.  By  adding  the  additional  data  sources,  a  more  effective 
range  of  estimates  could  be  achieved.  Another  possibility  is  to  see  if  the  costs  are 
increased  or  decreased  as  programs  are  delayed  in  the  process.  Such  as  when  a  program 
doesn’t  pass  a  KDP  and  must  go  through  the  IPA  process  again. 

This  research  only  scratches  the  surface  on  the  potential  research  trying  to  capture 
the  cost  of  oversight.  These  funds  aren’t  always  budgeted  dollars,  but  do  cost  the 
government  in  direct  or  indirect  costs.  Overall,  this  research  provides  the  basis  for  the 
identification  for  potential  cost  savings  in  the  acquisition  environment  to  help  slow  the 
increasing  costs  in  major  weapons  system  procurement. 
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